This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 06db44a3b3d [SPARK-44487][TEST] Fix KubernetesSuite report NPE when
not set spark.kubernetes.test.unpackSparkDir
06db44a3b3d is described below
commit 06db44a3b3d9a7a4ad1c4ad17d559a4cd152ce53
Author: Jia Fan <[email protected]>
AuthorDate: Fri Jul 21 10:47:48 2023 +0900
[SPARK-44487][TEST] Fix KubernetesSuite report NPE when not set
spark.kubernetes.test.unpackSparkDir
### What changes were proposed in this pull request?
Fix KubernetesSuite report NPE when not set
`spark.kubernetes.test.unpackSparkDir`
```java
Exception encountered when invoking run on a nested suite.
java.lang.NullPointerException
at sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:77)
at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
at java.nio.file.Paths.get(Paths.java:84)
at
org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.$anonfun$beforeAll$4(KubernetesSuite.scala:164)
at
org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.$anonfun$beforeAll$4$adapted(KubernetesSuite.scala:163)
at
scala.collection.LinearSeqOptimized.find(LinearSeqOptimized.scala:115)
at
scala.collection.LinearSeqOptimized.find$(LinearSeqOptimized.scala:112)
at scala.collection.immutable.List.find(List.scala:91)
at
org.apache.spark.deploy.k8s.integrationtest.KubernetesSuite.beforeAll(KubernetesSuite.scala:163)
```
### Why are the changes needed?
`spark.kubernetes.test.unpackSparkDir` is an option, not necessary config.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
exist test. Tested in local
Closes #42081 from Hisoka-X/SPARK-44487_k8s_test_npe.
Authored-by: Jia Fan <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
index f52af87a745..33485f2d6bc 100644
---
a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
+++
b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
@@ -160,7 +160,7 @@ class KubernetesSuite extends SparkFunSuite
// Try the spark test home
sys.props("spark.test.home")
)
- val sparkDirProp = possible_spark_dirs.find(x =>
+ val sparkDirProp = possible_spark_dirs.filter(_ != null).find(x =>
new File(Paths.get(x).toFile, "bin/spark-submit").exists).orNull
require(sparkDirProp != null,
s"Spark home directory must be provided in system properties tested
$possible_spark_dirs")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
