This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 53643924459 [SPARK-45670][CORE] SparkSubmit does not support
`--total-executor-cores` when deploying on K8s
53643924459 is described below
commit 536439244593d40bdab88e9d3657f2691d3d33f2
Author: Cheng Pan <[email protected]>
AuthorDate: Thu Oct 26 09:54:34 2023 -0700
[SPARK-45670][CORE] SparkSubmit does not support `--total-executor-cores`
when deploying on K8s
### What changes were proposed in this pull request?
Remove Kubernetes from the support list of `--total-executor-cores` in
SparkSubmit
### Why are the changes needed?
`--total-executor-cores` does not take effect in Spark on K8s, [the
comments from original
PR](https://github.com/apache/spark/pull/19717#discussion_r154568773) also
proves that
### Does this PR introduce _any_ user-facing change?
The output of `spark-submit --help` changed
```patch
...
- Spark standalone and Kubernetes only:
+ Spark standalone only:
--total-executor-cores NUM Total cores for all executors.
...
```
### How was this patch tested?
Pass GA and review.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #43536 from pan3793/tec.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala | 2 +-
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
index f101a180ea5..883ca62ae22 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
@@ -674,7 +674,7 @@ private[spark] class SparkSubmit extends Logging {
confKey = EXECUTOR_CORES.key),
OptionAssigner(args.executorMemory, STANDALONE | YARN | KUBERNETES,
ALL_DEPLOY_MODES,
confKey = EXECUTOR_MEMORY.key),
- OptionAssigner(args.totalExecutorCores, STANDALONE | KUBERNETES,
ALL_DEPLOY_MODES,
+ OptionAssigner(args.totalExecutorCores, STANDALONE, ALL_DEPLOY_MODES,
confKey = CORES_MAX.key),
OptionAssigner(args.files, LOCAL | STANDALONE | KUBERNETES,
ALL_DEPLOY_MODES,
confKey = FILES.key),
diff --git
a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
index bd6507c8ac4..534c1400061 100644
--- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
@@ -566,7 +566,7 @@ private[deploy] class SparkSubmitArguments(args:
Seq[String], env: Map[String, S
| --kill SUBMISSION_ID If given, kills the driver specified.
| --status SUBMISSION_ID If given, requests the status of the
driver specified.
|
- | Spark standalone and Kubernetes only:
+ | Spark standalone only:
| --total-executor-cores NUM Total cores for all executors.
|
| Spark standalone, YARN and Kubernetes only:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
