Hi Team, We have observed a similar issue with the Kubernetes API server master URL and would like to share our findings and proposed fix to get your feedback. Problem When the driver pod runs inside a Kubernetes cluster, Spark creates a Fabric8 Kubernetes client via SparkKubernetesClientFactory.createKubernetesClient(). The factory first calls autoConfigure(), which correctly picks up the in-cluster KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables and sets the master URL to an IP address (for example, https://10.96.0.1:443). This avoids the need for DNS resolution. However, Spark then unconditionally overrides this configuration with .withMasterUrl(master), where master is the default spark.kubernetes.driver.master value:
val KUBERNETES_MASTER_INTERNAL_URL = "https://kubernetes.default.svc"; Because kubernetes.default.svc has only two dots, clusters with an ndots:5 resolver configuration generate multiple NXDOMAIN responses before the name is resolved. Additionally, Fabric8's IpAddressMatcher internally calls InetAddress.getByName("kubernetes.default.svc"), which results in further DNS lookups and NXDOMAIN responses. Proposed Fix In SparkKubernetesClientFactory.scala, we only override the master URL when running outside the cluster or when a custom master URL has been explicitly provided. When the default https://kubernetes.default.svc is used while running in-cluster, we allow Fabric8 to retain the IP-based URL obtained from KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT. The proposed change is: if (!sys.env.contains("KUBERNETES_SERVICE_HOST") || master != KUBERNETES_MASTER_INTERNAL_URL) { baseConfigBuilder.withMasterUrl(master) } else { baseConfigBuilder } This preserves user-provided spark.master / spark.kubernetes.driver.master values for cross-cluster or external submission scenarios, while eliminating unnecessary DNS lookups for the default in-cluster case. Questions 1. Is this the correct approach, or is there a preferred way to make Spark use the in-cluster Kubernetes API server IP? 2. Has this issue been addressed in newer Spark versions, or is there an existing JIRA tracking it? We are currently running Spark 3.5.2 and have verified the proposed fix in our environment. Looking forward to your guidance. Best regards, Shazeb Khan ________________________________ From: Dongjoon Hyun <[email protected]> Sent: Tuesday, August 25, 2026 21:04 To: [email protected] <[email protected]> Subject: Re: [apache-spark] [K8S] Multiple Failed DNS Lookups When Resolving the Driver Hostname in Spark Applications on Kubernetes [You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Please try to use the existing feature, spark.kubernetes.executor.useDriverPodIP , first. FYI, Apache Spark 4.1 delivered many improvements under the umbrella JIRA issue, SPARK-54016 (Improve K8s support in Spark 4.1.0). Specifically for the mentioned issue, I can say that Apache Spark community already moved to allow executors to use Driver pod's IP directly via the following two step-wised improvements. SPARK-53944 Support spark.kubernetes.executor.useDriverPodIP (Spark 4.1.0) SPARK-57351 Enable `spark.kubernetes.executor.useDriverPodIP` by default (Spark 4.2.0) Dongjoon. On 2026/08/24 11:50:37 Uroš Bojanić wrote: > Hi Shazeb, > > Thank you for your investigation and the detailed write up here. > > +1 to fixing this. I would lean toward the trailing-dot FQDN (configurable > domain) as the robust form, with the bare service name as an acceptable > minimal fix given the co-namespaced executor model. > > In any case, please feel free to file a regular JIRA for this (no SPIP needed > for this chagne) under K8s, and eventually ping committers (e.g. Dongjoon > Hyun, Yang Jie) for further technical discussion on the PR! > > Best, > Uroš > > On 2026/08/24 11:21:47 Shazeb Khan via dev wrote: > > Hi Spark Community, > > I am writing regarding a DNS resolution issue we observed in our production > > Kubernetes environment while running multiple concurrent Spark pipelines > > using spark-submit. We are experiencing slowness because each time an > > executor attempts to connect to the driver, two unnecessary DNS queries > > fail before the driver hostname is successfully resolved. We need a most > > appropriate solution for this issue. > > Environment > > > > * > > Apache Spark: v3.5.2 & v4.1.1 > > * > > Deployment Mode: Cluster mode on Kubernetes > > * > > CoreDNS: Standard Kubernetes configuration > > * > > Pod DNS resolver: ndots:5 > > * > > Default DNS search domains: > > > > search default.svc.cluster.local svc.cluster.local cluster.local > > > > > > Current Behaviour > > DriverServiceFeatureStep.scala currently constructs spark.driver.host as: > > > > val driverHostname = s"$resolvedServiceName.${kubernetesConf.namespace}.svc" > > > > For example: > > > > sample-app-driver-svc.default.svc > > > > Because this string does not end with a dot, the operating system inside > > the Pod treats it as a relative domain name. > > Due to the default Kubernetes setting of ndots:5, any domain with fewer > > than 5 dots is passed through the local search paths first. > > sample-app-driver-svc.default.svc contains exactly 2 dots, so it triggers > > sequential search path appends: > > > > Attempt 1: > > sample-app-driver-svc.default.svc.default.svc.cluster.local. → NXDOMAIN > > > > Attempt 2: > > sample-app-driver-svc.default.svc.svc.cluster.local. → NXDOMAIN > > > > Attempt 3: > > sample-app-driver-svc.default.svc.cluster.local. → NOERROR > > > > This results in unnecessary DNS queries before the driver service is > > resolved. > > > > Proposed Solution - Use Service Name Only > > We propose setting spark.driver.host to only the Kubernetes Resolved > > Service name: > > > > sample-app-driver-svc > > > > With the service name containing no dots, the resolver can directly use the > > Kubernetes DNS search path: > > > > sample-app-driver-svc.default.svc.cluster.local. > > > > This allows the driver service to resolve on the first DNS lookup without > > generating the intermediate NXDOMAIN queries. > > This approach relies on the standard Kubernetes DNS search domains > > configured in the pod and avoids embedding the namespace and svc components > > directly into spark.driver.host. > > > > CoreDNS Evidence - Verified in Production > > With the proposed service-name-only approach, the DNS request is resolved > > directly: > > > > [INFO] "A IN sample-app-driver-svc.default.svc.cluster.local." NOERROR > > [INFO] "AAAA IN sample-app-driver-svc.default.svc.cluster.local." NOERROR > > > > With the current approach, CoreDNS receives the following queries before > > successfully resolving the service: > > > > [INFO] "A IN > > sample-app-driver-svc.default.svc.default.svc.cluster.local." NXDOMAIN > > [INFO] "AAAA IN > > sample-app-driver-svc.default.svc.default.svc.cluster.local." NXDOMAIN > > > > [INFO] "A IN sample-app-driver-svc.default.svc.svc.cluster.local." > > NXDOMAIN > > [INFO] "AAAA IN sample-app-driver-svc.default.svc.svc.cluster.local." > > NXDOMAIN > > > > [INFO] "A IN sample-app-driver-svc.default.svc.cluster.local." NOERROR > > [INFO] "AAAA IN sample-app-driver-svc.default.svc.cluster.local." NOERROR > > > > Impact on Multi-Pipeline Deployments > > In our deployment, we run multiple Spark pipelines simultaneously. Each > > executor startup generates unnecessary failed DNS queries. > > At scale, this creates additional load on CoreDNS, which is a shared > > cluster-wide component, due to avoidable NXDOMAIN lookups. > > The comparison is: > > Approach > > NXDOMAIN Queries > > Resolves On > > Current .default.svc > > 4 > > 3rd attempt > > Service name only (proposed) > > 0 > > 1st attempt > > > > > > Proposed Code Change > > Current: > > > > val driverHostname = s"$resolvedServiceName.${kubernetesConf.namespace}.svc" > > > > Proposed: > > > > val driverHostname = s"$resolvedServiceName" > > > > > > Questions to the Community: > > > > 1. > > Is there any way to resolve this Failed DNS lookup issue using External > > Properties / Some other options ? > > 2. > > Is there a specific reason why the hostname includes .namespace.svc rather > > than relying on the standard Kubernetes DNS search path? > > 3. > > Is it safe to go with above proposed solution, or could it break any > > existing behaviour? > > > > > > Thanks and Regards, > > > > Shazeb Khan > > > > ClearTrail Technologies Pvt. Ltd. > > > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
