This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.3 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push: new aa84c061728 [SPARK-42785][K8S][CORE] When spark submit without `--deploy-mode`, avoid facing NPE in Kubernetes Case aa84c061728 is described below commit aa84c06172851daba61a88fce9839466d2de55d0 Author: zwangsheng <2213335...@qq.com> AuthorDate: Tue Mar 14 08:49:13 2023 -0700 [SPARK-42785][K8S][CORE] When spark submit without `--deploy-mode`, avoid facing NPE in Kubernetes Case ### What changes were proposed in this pull request? After https://github.com/apache/spark/pull/37880 when user spark submit without `--deploy-mode XXX` or `–conf spark.submit.deployMode=XXXX`, may face NPE with this code. ### Why are the changes needed? https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#164 ```scala args.deployMode.equals("client") && ``` Of course, submit without `deployMode` is not allowed and will throw an exception and terminate the application, but we should leave it to the later logic to give the appropriate hint instead of giving a NPE. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? ![popo_2023-03-14 17-50-46](https://user-images.githubusercontent.com/52876270/224965310-ba9ec82f-e668-4a06-b6ff-34c3e80ca0b4.jpg) Closes #40414 from zwangsheng/SPARK-42785. Authored-by: zwangsheng <2213335...@qq.com> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> (cherry picked from commit 767253bb6219f775a8a21f1cdd0eb8c25fa0b9de) Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala index 76d17ef6be7..cf840879dcf 100644 --- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala +++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala @@ -161,7 +161,7 @@ private[spark] class SparkSubmit extends Logging { // deploy mode with k8s resource manager, the spark submit in the driver container // is done in client mode. val isKubernetesClusterModeDriver = args.master.startsWith("k8s") && - args.deployMode.equals("client") && + "client".equals(args.deployMode) && args.toSparkConf().getBoolean("spark.kubernetes.submitInDriver", false) if (isKubernetesClusterModeDriver) { logInfo("Running driver with proxy user. Cluster manager: Kubernetes") --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org