This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2bb8b200044d [SPARK-57351][K8S][CORE] Enable
`spark.kubernetes.executor.useDriverPodIP` by default
2bb8b200044d is described below
commit 2bb8b200044d96adf13565ef493736e39ce13f4e
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Jun 9 14:24:32 2026 -0700
[SPARK-57351][K8S][CORE] Enable `spark.kubernetes.executor.useDriverPodIP`
by default
### What changes were proposed in this pull request?
This PR sets the default value of
`spark.kubernetes.executor.useDriverPodIP` to `true` at Apache Spark 4.3.0.
### Why are the changes needed?
Introduced in SPARK-53944 (4.1.0), this option lets executor pods connect
to the driver pod IP directly instead of the driver's Kubernetes `Service`,
bypassing the known K8s DNS issue. Since a Spark driver pod is not restarted
during an application, its IP is stable, making this a safe default that avoids
DNS lookups.
- https://github.com/apache/spark/pull/52650
### Does this PR introduce _any_ user-facing change?
Yes. By default, executor pods now connect to the driver via the driver pod
IP instead of the driver `Service` DNS name. To restore the legacy behavior,
set `spark.kubernetes.executor.useDriverPodIP` to `false`.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #56412 from dongjoon-hyun/SPARK-57351.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
core/src/main/scala/org/apache/spark/SparkContext.scala | 2 +-
docs/core-migration-guide.md | 4 ++++
docs/running-on-kubernetes.md | 2 +-
.../core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala | 2 +-
4 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala
b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 0262144490ce..e00e34b89b84 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -455,7 +455,7 @@ class SparkContext(config: SparkConf) extends Logging {
// Set Spark driver host and port system properties. This explicitly sets
the configuration
// instead of relying on the default value of the config constant.
if (SparkMasterRegex.isK8s(master) &&
- _conf.getBoolean("spark.kubernetes.executor.useDriverPodIP", false)) {
+ _conf.getBoolean("spark.kubernetes.executor.useDriverPodIP", true)) {
logInfo("Use DRIVER_BIND_ADDRESS instead of DRIVER_HOST_ADDRESS as
driver address " +
"because spark.kubernetes.executor.useDriverPodIP is true in K8s
mode.")
_conf.set(DRIVER_HOST_ADDRESS, _conf.get(DRIVER_BIND_ADDRESS))
diff --git a/docs/core-migration-guide.md b/docs/core-migration-guide.md
index 284cd7a8639b..9be60572387c 100644
--- a/docs/core-migration-guide.md
+++ b/docs/core-migration-guide.md
@@ -22,6 +22,10 @@ license: |
* Table of contents
{:toc}
+## Upgrading from Core 4.2 to 4.3
+
+- Since Spark 4.3, Spark executor pods connect to the driver via the driver
pod IP directly instead of the driver's Kubernetes Service. To restore the
legacy behavior, you can set `spark.kubernetes.executor.useDriverPodIP` to
`false`.
+
## Upgrading from Core 4.1 to 4.2
- Since Spark 4.2, Spark Master REST API uses Java 21 virtual threads by
default when running on Java 21 or later. To restore the legacy behavior, you
can set `spark.master.rest.virtualThread.enabled` to `false`.
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index aa753e259bcc..9be54d898486 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1585,7 +1585,7 @@ See the [configuration page](configuration.html) for
information on Spark config
</tr>
<tr>
<td><code>spark.kubernetes.executor.useDriverPodIP</code></td>
- <td><code>false</code></td>
+ <td><code>true</code></td>
<td>
If true, executor pods use Driver pod IP directly instead of Driver
Service.
</td>
diff --git
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
index 0e79cd0d5606..120881469e3a 100644
---
a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
+++
b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
@@ -125,7 +125,7 @@ private[spark] object Config extends Logging {
.doc("If true, executor pods use Driver pod IP directly instead of
Driver Service.")
.version("4.1.0")
.booleanConf
- .createWithDefault(false)
+ .createWithDefault(true)
val KUBERNETES_NAMESPACE =
ConfigBuilder("spark.kubernetes.namespace")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
