This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-kubernetes-operator.git
The following commit(s) were added to refs/heads/main by this push:
new 4a26f3b [SPARK-57350] Enable
`spark.kubernetes.executor.useDriverPodIP` by default
4a26f3b is described below
commit 4a26f3b26743440fae33223b87f6104a8ed11caa
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Jun 9 11:48:45 2026 -0700
[SPARK-57350] Enable `spark.kubernetes.executor.useDriverPodIP` by default
### What changes were proposed in this pull request?
This PR sets `spark.kubernetes.executor.useDriverPodIP` to `true` by
default via
`setIfMissing` in `SparkAppSubmissionWorker#buildDriverConf`, next to the
other
operator-injected defaults. Since it uses `setIfMissing`, an explicit value
in the
application spec is always respected.
### Why are the changes needed?
`spark.kubernetes.executor.useDriverPodIP` (added in Spark 4.1.0 via
SPARK-53944) makes
executor pods connect to the driver using the driver pod IP instead of the
driver
`Service` hostname, bypassing Kubernetes DNS. The operator fully manages
the driver pod
lifecycle, and in cluster mode the driver is not restarted (the application
terminates if
it dies), so the pod IP is stable for the application's lifetime. Enabling
it by default
avoids DNS propagation delays/failures on executor startup and reduces
CoreDNS load. It is
a no-op for Spark versions older than 4.1.0.
- https://github.com/apache/spark/pull/52650
### Does this PR introduce _any_ user-facing change?
Yes. When `spark.kubernetes.executor.useDriverPodIP` is unset, the operator
now defaults
it to `true` (previously Spark's default of `false`). Applications that set
it explicitly
are unaffected.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #702 from dongjoon-hyun/SPARK-57350.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../java/org/apache/spark/k8s/operator/SparkAppSubmissionWorker.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkAppSubmissionWorker.java
b/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkAppSubmissionWorker.java
index 8c048c9..7bf8361 100644
---
a/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkAppSubmissionWorker.java
+++
b/spark-submission-worker/src/main/java/org/apache/spark/k8s/operator/SparkAppSubmissionWorker.java
@@ -168,6 +168,7 @@ public class SparkAppSubmissionWorker {
effectiveSparkConf.setIfMissing("spark.authenticate", "true");
effectiveSparkConf.setIfMissing("spark.io.encryption.enabled", "true");
effectiveSparkConf.setIfMissing("spark.kubernetes.driver.annotateExitException",
"true");
+
effectiveSparkConf.setIfMissing("spark.kubernetes.executor.useDriverPodIP",
"true");
// In case of static allocation, use K8s Garbage Collection instead of
explicit API invocations
if
(!"true".equalsIgnoreCase(effectiveSparkConf.get("spark.dynamicAllocation.enabled",
"false"))
&&
applicationSpec.getApplicationTolerations().getResourceRetainPolicy() !=
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
