Hi Shazeb,
Thank you for your investigation and the detailed write up here.
+1 to fixing this. I would lean toward the trailing-dot FQDN (configurable
domain) as the robust form, with the bare service name as an acceptable minimal
fix given the co-namespaced executor model.
In any case, please feel free to file a regular JIRA for this (no SPIP needed
for this chagne) under K8s, and eventually ping committers (e.g. Dongjoon Hyun,
Yang Jie) for further technical discussion on the PR!
Best,
Uroš
On 2026/08/24 11:21:47 Shazeb Khan via dev wrote:
> Hi Spark Community,
> I am writing regarding a DNS resolution issue we observed in our production
> Kubernetes environment while running multiple concurrent Spark pipelines
> using spark-submit. We are experiencing slowness because each time an
> executor attempts to connect to the driver, two unnecessary DNS queries fail
> before the driver hostname is successfully resolved. We need a most
> appropriate solution for this issue.
> Environment
>
> *
> Apache Spark: v3.5.2 & v4.1.1
> *
> Deployment Mode: Cluster mode on Kubernetes
> *
> CoreDNS: Standard Kubernetes configuration
> *
> Pod DNS resolver: ndots:5
> *
> Default DNS search domains:
>
> search default.svc.cluster.local svc.cluster.local cluster.local
>
>
> Current Behaviour
> DriverServiceFeatureStep.scala currently constructs spark.driver.host as:
>
> val driverHostname = s"$resolvedServiceName.${kubernetesConf.namespace}.svc"
>
> For example:
>
> sample-app-driver-svc.default.svc
>
> Because this string does not end with a dot, the operating system inside the
> Pod treats it as a relative domain name.
> Due to the default Kubernetes setting of ndots:5, any domain with fewer than
> 5 dots is passed through the local search paths first.
> sample-app-driver-svc.default.svc contains exactly 2 dots, so it triggers
> sequential search path appends:
>
> Attempt 1:
> sample-app-driver-svc.default.svc.default.svc.cluster.local. → NXDOMAIN
>
> Attempt 2:
> sample-app-driver-svc.default.svc.svc.cluster.local. → NXDOMAIN
>
> Attempt 3:
> sample-app-driver-svc.default.svc.cluster.local. → NOERROR
>
> This results in unnecessary DNS queries before the driver service is resolved.
>
> Proposed Solution - Use Service Name Only
> We propose setting spark.driver.host to only the Kubernetes Resolved Service
> name:
>
> sample-app-driver-svc
>
> With the service name containing no dots, the resolver can directly use the
> Kubernetes DNS search path:
>
> sample-app-driver-svc.default.svc.cluster.local.
>
> This allows the driver service to resolve on the first DNS lookup without
> generating the intermediate NXDOMAIN queries.
> This approach relies on the standard Kubernetes DNS search domains configured
> in the pod and avoids embedding the namespace and svc components directly
> into spark.driver.host.
>
> CoreDNS Evidence - Verified in Production
> With the proposed service-name-only approach, the DNS request is resolved
> directly:
>
> [INFO] "A IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
> [INFO] "AAAA IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
>
> With the current approach, CoreDNS receives the following queries before
> successfully resolving the service:
>
> [INFO] "A IN sample-app-driver-svc.default.svc.default.svc.cluster.local."
> NXDOMAIN
> [INFO] "AAAA IN sample-app-driver-svc.default.svc.default.svc.cluster.local."
> NXDOMAIN
>
> [INFO] "A IN sample-app-driver-svc.default.svc.svc.cluster.local." NXDOMAIN
> [INFO] "AAAA IN sample-app-driver-svc.default.svc.svc.cluster.local." NXDOMAIN
>
> [INFO] "A IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
> [INFO] "AAAA IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
>
> Impact on Multi-Pipeline Deployments
> In our deployment, we run multiple Spark pipelines simultaneously. Each
> executor startup generates unnecessary failed DNS queries.
> At scale, this creates additional load on CoreDNS, which is a shared
> cluster-wide component, due to avoidable NXDOMAIN lookups.
> The comparison is:
> Approach
> NXDOMAIN Queries
> Resolves On
> Current .default.svc
> 4
> 3rd attempt
> Service name only (proposed)
> 0
> 1st attempt
>
>
> Proposed Code Change
> Current:
>
> val driverHostname = s"$resolvedServiceName.${kubernetesConf.namespace}.svc"
>
> Proposed:
>
> val driverHostname = s"$resolvedServiceName"
>
>
> Questions to the Community:
>
> 1.
> Is there any way to resolve this Failed DNS lookup issue using External
> Properties / Some other options ?
> 2.
> Is there a specific reason why the hostname includes .namespace.svc rather
> than relying on the standard Kubernetes DNS search path?
> 3.
> Is it safe to go with above proposed solution, or could it break any existing
> behaviour?
>
>
> Thanks and Regards,
>
> Shazeb Khan
>
> ClearTrail Technologies Pvt. Ltd.
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
