This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 2cf8f025946 [SPARK-42785][K8S][CORE] When spark submit without
`--deploy-mode`, avoid facing NPE in Kubernetes Case
2cf8f025946 is described below
commit 2cf8f025946041116e1925703537e13a0c6bd709
Author: zwangsheng <[email protected]>
AuthorDate: Tue Mar 14 08:49:13 2023 -0700
[SPARK-42785][K8S][CORE] When spark submit without `--deploy-mode`, avoid
facing NPE in Kubernetes Case
### What changes were proposed in this pull request?
After https://github.com/apache/spark/pull/37880 when user spark submit
without `--deploy-mode XXX` or `–conf spark.submit.deployMode=XXXX`, may face
NPE with this code.
### Why are the changes needed?
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#164
```scala
args.deployMode.equals("client") &&
```
Of course, submit without `deployMode` is not allowed and will throw an
exception and terminate the application, but we should leave it to the later
logic to give the appropriate hint instead of giving a NPE.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Closes #40414 from zwangsheng/SPARK-42785.
Authored-by: zwangsheng <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 767253bb6219f775a8a21f1cdd0eb8c25fa0b9de)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index eff39255407..4e3aee595b2 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -161,7 +161,7 @@ private[spark] class SparkSubmit extends Logging {
// deploy mode with k8s resource manager, the spark submit in the
driver container
// is done in client mode.
val isKubernetesClusterModeDriver = args.master.startsWith("k8s") &&
- args.deployMode.equals("client") &&
+ "client".equals(args.deployMode) &&
args.toSparkConf().getBoolean("spark.kubernetes.submitInDriver",
false)
if (isKubernetesClusterModeDriver) {
logInfo("Running driver with proxy user. Cluster manager:
Kubernetes")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
