[spark] branch master updated: [SPARK-39261][CORE][FOLLOWUP] Improve newline formatting for error messages
This is an automated email from the ASF dual-hosted git repository.
yumwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 00277f5370c [SPARK-39261][CORE][FOLLOWUP] Improve newline formatting
for error messages
00277f5370c is described below
commit 00277f5370c56d207aae437293958d1b5e4470d9
Author: sychen
AuthorDate: Sun May 29 09:38:34 2022 +0800
[SPARK-39261][CORE][FOLLOWUP] Improve newline formatting for error messages
### What changes were proposed in this pull request?
Use `java.nio.file.Files.delete` instead of
`org.apache.commons.io.FileUtils#delete`
### Why are the changes needed?
`org.apache.commons.io.FileUtils#delete` is a method only available in
version 2.9, hadoop2 uses version 2.4 of commons-io.
Build failed
```
./dev/make-distribution.sh --tgz -Pyarn -Phive -Phive-thriftserver
-Phadoop-2 -DskipTests -Dmaven.javadoc.skip=true
```
```
spark/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala:92:
value delete is not a member of object org.apache.commons.io.FileUtils
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
exist UT
Closes #36710 from cxzl25/SPARK-39261-FOLLOWUP.
Authored-by: sychen
Signed-off-by: Yuming Wang
---
core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala
b/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala
index 6ad80950931..9092571b9b9 100644
--- a/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala
+++ b/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala
@@ -19,6 +19,7 @@ package org.apache.spark
import java.io.File
import java.nio.charset.StandardCharsets
+import java.nio.file.Files
import java.util.IllegalFormatException
import com.fasterxml.jackson.annotation.JsonInclude.Include
@@ -89,7 +90,7 @@ class SparkThrowableSuite extends SparkFunSuite {
if (rewrittenString.trim != errorClassFileContents.trim) {
val errorClassesFile = new File(errorClassDir, new
File(errorClassesUrl.getPath).getName)
logInfo(s"Regenerating error class file $errorClassesFile")
-FileUtils.delete(errorClassesFile)
+Files.delete(errorClassesFile.toPath)
FileUtils.writeStringToFile(errorClassesFile, rewrittenString,
StandardCharsets.UTF_8)
}
} else {
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-39282][SQL] Replace If-Else branch with bitwise operators in roundNumberOfBytesToNearestWord
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b12c12790c7 [SPARK-39282][SQL] Replace If-Else branch with bitwise
operators in roundNumberOfBytesToNearestWord
b12c12790c7 is described below
commit b12c12790c7c4453c1f9e62d49feba30957c0af3
Author: zhixingheyi-tian
AuthorDate: Sat May 28 14:15:59 2022 -0500
[SPARK-39282][SQL] Replace If-Else branch with bitwise operators in
roundNumberOfBytesToNearestWord
### What changes were proposed in this pull request?
Use bitwise operators to avoid If-Else branch and improve computation
performance.
### How was this patch tested?
Existed UTs.
Closes #36659 from zhixingheyi-tian/avoid_ifelse_branch.
Authored-by: zhixingheyi-tian
Signed-off-by: Sean Owen
---
.../main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java | 6 +-
1 file changed, 1 insertion(+), 5 deletions(-)
diff --git
a/common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java
b/common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java
index deb7d2bf1b0..500bc9de325 100644
---
a/common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java
+++
b/common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java
@@ -39,11 +39,7 @@ public class ByteArrayMethods {
public static long roundNumberOfBytesToNearestWord(long numBytes) {
long remainder = numBytes & 0x07; // This is equivalent to `numBytes % 8`
-if (remainder == 0) {
- return numBytes;
-} else {
- return numBytes + (8 - remainder);
-}
+return numBytes + ((8 - remainder) & 0x7);
}
// Some JVMs can't allocate arrays of length Integer.MAX_VALUE; actual max
is somewhat smaller.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-39324][CORE] Log `ExecutorDecommission` as INFO level in `TaskSchedulerImpl`
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 3b8465b41ec [SPARK-39324][CORE] Log `ExecutorDecommission` as INFO level in `TaskSchedulerImpl` 3b8465b41ec is described below commit 3b8465b41ec3ecc1f97701e93f7043bf0cacb42e Author: Dongjoon Hyun AuthorDate: Sat May 28 10:38:16 2022 -0700 [SPARK-39324][CORE] Log `ExecutorDecommission` as INFO level in `TaskSchedulerImpl` ### What changes were proposed in this pull request? This PR aims to log `ExecutorDecommission` as INFO level in `TaskSchedulerImpl` in Apache Spark 3.4. ### Why are the changes needed? Like the other modules, `TaskSchedulerImpl` had better log the decommission as `INFO` level instead of `ERROR`. ``` 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission executors: 8 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8 to decommissioning. 22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers (BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning. 22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on 100.103.40.13: Executor decommission. 22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove reason statistics: ... 22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7) 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster. 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(8, 100.103.40.13, 43353, None) 22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in removeExecutor 22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8 (epoch 7) 22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested 22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster. 22/05/28 01:25:34 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 8 ``` ### Does this PR introduce _any_ user-facing change? No. This is a change on log level. ### How was this patch tested? Pass the CIs. Closes #36707 from dongjoon-hyun/SPARK-39324. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala index 55db73ab2a0..55938d8b77e 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala @@ -1018,6 +1018,8 @@ private[spark] class TaskSchedulerImpl( logDebug(s"Executor $executorId on $hostPort lost, but reason not yet known.") case ExecutorKilled => logInfo(s"Executor $executorId on $hostPort killed by driver.") +case _: ExecutorDecommission => + logInfo(s"Executor $executorId on $hostPort is decommissioned.") case _ => logError(s"Lost executor $executorId on $hostPort: $reason") } - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
