Please try to use the existing feature,
spark.kubernetes.executor.useDriverPodIP , first.
FYI, Apache Spark 4.1 delivered many improvements under the umbrella JIRA
issue, SPARK-54016 (Improve K8s support in Spark 4.1.0).
Specifically for the mentioned issue, I can say that Apache Spark community
already moved to allow executors to use Driver pod's IP directly via the
following two step-wised improvements.
SPARK-53944 Support spark.kubernetes.executor.useDriverPodIP (Spark 4.1.0)
SPARK-57351 Enable `spark.kubernetes.executor.useDriverPodIP` by default (Spark
4.2.0)
Dongjoon.
On 2026/08/24 11:50:37 Uroš Bojanić wrote:
> Hi Shazeb,
>
> Thank you for your investigation and the detailed write up here.
>
> +1 to fixing this. I would lean toward the trailing-dot FQDN (configurable
> domain) as the robust form, with the bare service name as an acceptable
> minimal fix given the co-namespaced executor model.
>
> In any case, please feel free to file a regular JIRA for this (no SPIP needed
> for this chagne) under K8s, and eventually ping committers (e.g. Dongjoon
> Hyun, Yang Jie) for further technical discussion on the PR!
>
> Best,
> Uroš
>
> On 2026/08/24 11:21:47 Shazeb Khan via dev wrote:
> > Hi Spark Community,
> > I am writing regarding a DNS resolution issue we observed in our production
> > Kubernetes environment while running multiple concurrent Spark pipelines
> > using spark-submit. We are experiencing slowness because each time an
> > executor attempts to connect to the driver, two unnecessary DNS queries
> > fail before the driver hostname is successfully resolved. We need a most
> > appropriate solution for this issue.
> > Environment
> >
> > *
> > Apache Spark: v3.5.2 & v4.1.1
> > *
> > Deployment Mode: Cluster mode on Kubernetes
> > *
> > CoreDNS: Standard Kubernetes configuration
> > *
> > Pod DNS resolver: ndots:5
> > *
> > Default DNS search domains:
> >
> > search default.svc.cluster.local svc.cluster.local cluster.local
> >
> >
> > Current Behaviour
> > DriverServiceFeatureStep.scala currently constructs spark.driver.host as:
> >
> > val driverHostname = s"$resolvedServiceName.${kubernetesConf.namespace}.svc"
> >
> > For example:
> >
> > sample-app-driver-svc.default.svc
> >
> > Because this string does not end with a dot, the operating system inside
> > the Pod treats it as a relative domain name.
> > Due to the default Kubernetes setting of ndots:5, any domain with fewer
> > than 5 dots is passed through the local search paths first.
> > sample-app-driver-svc.default.svc contains exactly 2 dots, so it triggers
> > sequential search path appends:
> >
> > Attempt 1:
> > sample-app-driver-svc.default.svc.default.svc.cluster.local. → NXDOMAIN
> >
> > Attempt 2:
> > sample-app-driver-svc.default.svc.svc.cluster.local. → NXDOMAIN
> >
> > Attempt 3:
> > sample-app-driver-svc.default.svc.cluster.local. → NOERROR
> >
> > This results in unnecessary DNS queries before the driver service is
> > resolved.
> >
> > Proposed Solution - Use Service Name Only
> > We propose setting spark.driver.host to only the Kubernetes Resolved
> > Service name:
> >
> > sample-app-driver-svc
> >
> > With the service name containing no dots, the resolver can directly use the
> > Kubernetes DNS search path:
> >
> > sample-app-driver-svc.default.svc.cluster.local.
> >
> > This allows the driver service to resolve on the first DNS lookup without
> > generating the intermediate NXDOMAIN queries.
> > This approach relies on the standard Kubernetes DNS search domains
> > configured in the pod and avoids embedding the namespace and svc components
> > directly into spark.driver.host.
> >
> > CoreDNS Evidence - Verified in Production
> > With the proposed service-name-only approach, the DNS request is resolved
> > directly:
> >
> > [INFO] "A IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
> > [INFO] "AAAA IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
> >
> > With the current approach, CoreDNS receives the following queries before
> > successfully resolving the service:
> >
> > [INFO] "A IN
> > sample-app-driver-svc.default.svc.default.svc.cluster.local." NXDOMAIN
> > [INFO] "AAAA IN
> > sample-app-driver-svc.default.svc.default.svc.cluster.local." NXDOMAIN
> >
> > [INFO] "A IN sample-app-driver-svc.default.svc.svc.cluster.local."
> > NXDOMAIN
> > [INFO] "AAAA IN sample-app-driver-svc.default.svc.svc.cluster.local."
> > NXDOMAIN
> >
> > [INFO] "A IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
> > [INFO] "AAAA IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
> >
> > Impact on Multi-Pipeline Deployments
> > In our deployment, we run multiple Spark pipelines simultaneously. Each
> > executor startup generates unnecessary failed DNS queries.
> > At scale, this creates additional load on CoreDNS, which is a shared
> > cluster-wide component, due to avoidable NXDOMAIN lookups.
> > The comparison is:
> > Approach
> > NXDOMAIN Queries
> > Resolves On
> > Current .default.svc
> > 4
> > 3rd attempt
> > Service name only (proposed)
> > 0
> > 1st attempt
> >
> >
> > Proposed Code Change
> > Current:
> >
> > val driverHostname = s"$resolvedServiceName.${kubernetesConf.namespace}.svc"
> >
> > Proposed:
> >
> > val driverHostname = s"$resolvedServiceName"
> >
> >
> > Questions to the Community:
> >
> > 1.
> > Is there any way to resolve this Failed DNS lookup issue using External
> > Properties / Some other options ?
> > 2.
> > Is there a specific reason why the hostname includes .namespace.svc rather
> > than relying on the standard Kubernetes DNS search path?
> > 3.
> > Is it safe to go with above proposed solution, or could it break any
> > existing behaviour?
> >
> >
> > Thanks and Regards,
> >
> > Shazeb Khan
> >
> > ClearTrail Technologies Pvt. Ltd.
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [email protected]
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
