[
https://issues.apache.org/jira/browse/SPARK-58967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18107890#comment-18107890
]
Shazeb Khan commented on SPARK-58967:
-------------------------------------
_+The Spark team suggested using an FQDN with a trailing dot ({{{}.{}}}) as the
more robust solution for this issue.+_
!image-2026-08-25-18-28-11-650.png|width=651,height=244!
*{color:#00875a}We have implemented the suggested approach and raised a PR for
the same:{color}*
[https://github.com/apache/spark/pull/58283]
Could you please let us know the expected timeline for this PR to be reviewed
and merged?
We would also like to understand which Spark release will include this fix once
the PR is merged. We are currently using Spark versions {*}3.5.2 and 4.1.1{*}.
Additionally, after the PR is merged, will the fix be available through a new
Spark binary/release, or is there another recommended way for us to consume the
change?
> [apache-spark] [K8S] Multiple Failed DNS Lookups When Resolving the Driver
> Hostname in Spark Applications on Kubernetes
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58967
> URL: https://issues.apache.org/jira/browse/SPARK-58967
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 3.5.2, 4.1.1
> Reporter: Shazeb Khan
> Priority: Major
> Labels: CoreDNS, kubernetes, pull-request-available, spark
> Attachments: image-2026-08-25-18-28-11-650.png
>
>
> We observed unnecessary DNS lookups when running multiple concurrent Spark
> pipelines using {{spark-submit}} in Kubernetes cluster mode.
> The issue occurs when executors attempt to connect to the Spark driver. The
> driver hostname generated by {{DriverServiceFeatureStep}} contains the
> namespace and {{svc}} components:
> {{sample-app-driver-svc.default.svc}}
> Because this hostname contains fewer dots than the configured {{ndots}} value
> ({{{}ndots:5{}}}), the pod's DNS resolver first attempts to resolve the
> hostname using the configured search domains.
> This results in two failed DNS lookups ({{{}NXDOMAIN{}}}) before the driver
> service is successfully resolved.
> h2. Environment
> * *Apache Spark:* 3.5.2 and 4.1.1
> * *Deployment Mode:* Cluster mode on Kubernetes
> * *CoreDNS:* Standard Kubernetes configuration
> * *Pod DNS resolver:* {{ndots:5}}
> * *DNS search domains:*
> {{search default.svc.cluster.local svc.cluster.local cluster.local}}
> h2. Current Behaviour
> {{DriverServiceFeatureStep.scala}} currently constructs {{spark.driver.host}}
> as:
> {{val driverHostname =
> s"$resolvedServiceName.${kubernetesConf.namespace}.svc"}}
> For example:
> {{sample-app-driver-svc.default.svc}}
> The DNS resolver then performs the following lookups:
> {{Attempt 1:
> sample-app-driver-svc.default.svc.default.svc.cluster.local. → NXDOMAIN
> Attempt 2:
> sample-app-driver-svc.default.svc.svc.cluster.local. → NXDOMAIN
> Attempt 3:
> sample-app-driver-svc.default.svc.cluster.local. → NOERROR}}
> As a result, two unnecessary DNS lookups are performed before the driver
> service is resolved.
> h2. Impact
> We are running multiple Spark pipelines concurrently in our production
> Kubernetes environment.
> Each executor startup can generate these unnecessary failed DNS queries while
> attempting to connect to the driver. With a large number of concurrent
> pipelines and executors, this results in additional DNS traffic and can
> contribute to delays during executor-to-driver connection establishment.
> It also generates additional {{NXDOMAIN}} traffic to CoreDNS, which is a
> shared cluster-wide component.
> h2. Observed DNS Queries
> With the current hostname construction, CoreDNS receives:
>
> {{[INFO] "A IN sample-app-driver-svc.default.svc.default.svc.cluster.local."
> NXDOMAIN
> [INFO] "AAAA IN sample-app-driver-svc.default.svc.default.svc.cluster.local."
> NXDOMAIN
> [INFO] "A IN sample-app-driver-svc.default.svc.svc.cluster.local." NXDOMAIN
> [INFO] "AAAA IN sample-app-driver-svc.default.svc.svc.cluster.local." NXDOMAIN
> [INFO] "A IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
> [INFO] "AAAA IN sample-app-driver-svc.default.svc.cluster.local." NOERROR}}
> h2. Proposed Solution
> One possible solution we have tested is to use only the Kubernetes Service
> name as {{{}spark.driver.host{}}}:
> {{sample-app-driver-svc}}
> The Kubernetes DNS search path would then resolve it directly as:
> {{sample-app-driver-svc.default.svc.cluster.local.}}
> This avoids the intermediate {{NXDOMAIN}} lookups and allows the service to
> be resolved on the first DNS attempt.
> The corresponding code change would be:
> *Current:*
> {{val driverHostname =
> s"$resolvedServiceName.${kubernetesConf.namespace}.svc"}}
> *Proposed:*
> {{val driverHostname = s"$resolvedServiceName"}}
> h2. Questions
> Before proposing a code change to {{{}DriverServiceFeatureStep{}}}, we would
> like to understand whether there is an existing Spark configuration or
> external property that can address this behaviour.
> Specifically:
> 1. Is there any Spark configuration/property that can be used to avoid these
> unnecessary DNS lookups without modifying {{{}DriverServiceFeatureStep{}}}?
> 2. Is there a specific reason why {{DriverServiceFeatureStep}} constructs
> {{spark.driver.host}} using {{.namespace.svc}} instead of relying on the
> standard Kubernetes DNS search path?
> 3. Would using only the Kubernetes Service name as {{spark.driver.host}} be a
> safe and supported approach for Kubernetes cluster mode?
> 4. If there is no existing configuration option, would changing the default
> hostname construction to use the Service name only be considered an
> appropriate fix?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
