Hi Spark Community,
I am writing regarding a DNS resolution issue we observed in our production
Kubernetes environment while running multiple concurrent Spark pipelines using
spark-submit. We are experiencing slowness because each time an executor
attempts to connect to the driver, two unnecessary DNS queries fail before the
driver hostname is successfully resolved. We need a most appropriate solution
for this issue.
Environment
*
Apache Spark: v3.5.2 & v4.1.1
*
Deployment Mode: Cluster mode on Kubernetes
*
CoreDNS: Standard Kubernetes configuration
*
Pod DNS resolver: ndots:5
*
Default DNS search domains:
search default.svc.cluster.local svc.cluster.local cluster.local
Current Behaviour
DriverServiceFeatureStep.scala currently constructs spark.driver.host as:
val driverHostname = s"$resolvedServiceName.${kubernetesConf.namespace}.svc"
For example:
sample-app-driver-svc.default.svc
Because this string does not end with a dot, the operating system inside the
Pod treats it as a relative domain name.
Due to the default Kubernetes setting of ndots:5, any domain with fewer than 5
dots is passed through the local search paths first.
sample-app-driver-svc.default.svc contains exactly 2 dots, so it triggers
sequential search path appends:
Attempt 1:
sample-app-driver-svc.default.svc.default.svc.cluster.local. → NXDOMAIN
Attempt 2:
sample-app-driver-svc.default.svc.svc.cluster.local. → NXDOMAIN
Attempt 3:
sample-app-driver-svc.default.svc.cluster.local. → NOERROR
This results in unnecessary DNS queries before the driver service is resolved.
Proposed Solution - Use Service Name Only
We propose setting spark.driver.host to only the Kubernetes Resolved Service
name:
sample-app-driver-svc
With the service name containing no dots, the resolver can directly use the
Kubernetes DNS search path:
sample-app-driver-svc.default.svc.cluster.local.
This allows the driver service to resolve on the first DNS lookup without
generating the intermediate NXDOMAIN queries.
This approach relies on the standard Kubernetes DNS search domains configured
in the pod and avoids embedding the namespace and svc components directly into
spark.driver.host.
CoreDNS Evidence - Verified in Production
With the proposed service-name-only approach, the DNS request is resolved
directly:
[INFO] "A IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
[INFO] "AAAA IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
With the current approach, CoreDNS receives the following queries before
successfully resolving the service:
[INFO] "A IN sample-app-driver-svc.default.svc.default.svc.cluster.local."
NXDOMAIN
[INFO] "AAAA IN sample-app-driver-svc.default.svc.default.svc.cluster.local."
NXDOMAIN
[INFO] "A IN sample-app-driver-svc.default.svc.svc.cluster.local." NXDOMAIN
[INFO] "AAAA IN sample-app-driver-svc.default.svc.svc.cluster.local." NXDOMAIN
[INFO] "A IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
[INFO] "AAAA IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
Impact on Multi-Pipeline Deployments
In our deployment, we run multiple Spark pipelines simultaneously. Each
executor startup generates unnecessary failed DNS queries.
At scale, this creates additional load on CoreDNS, which is a shared
cluster-wide component, due to avoidable NXDOMAIN lookups.
The comparison is:
Approach
NXDOMAIN Queries
Resolves On
Current .default.svc
4
3rd attempt
Service name only (proposed)
0
1st attempt
Proposed Code Change
Current:
val driverHostname = s"$resolvedServiceName.${kubernetesConf.namespace}.svc"
Proposed:
val driverHostname = s"$resolvedServiceName"
Questions to the Community:
1.
Is there any way to resolve this Failed DNS lookup issue using External
Properties / Some other options ?
2.
Is there a specific reason why the hostname includes .namespace.svc rather than
relying on the standard Kubernetes DNS search path?
3.
Is it safe to go with above proposed solution, or could it break any existing
behaviour?
Thanks and Regards,
Shazeb Khan
ClearTrail Technologies Pvt. Ltd.
