[
https://issues.apache.org/jira/browse/SPARK-58967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58967:
-----------------------------------
Labels: CoreDNS kubernetes pull-request-available spark (was: CoreDNS
kubernetes spark)
> [apache-spark] [K8S] Multiple Failed DNS Lookups When Resolving the Driver
> Hostname in Spark Applications on Kubernetes
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58967
> URL: https://issues.apache.org/jira/browse/SPARK-58967
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 3.5.2, 4.1.1
> Reporter: Shazeb Khan
> Priority: Major
> Labels: CoreDNS, kubernetes, pull-request-available, spark
>
> We observed unnecessary DNS lookups when running multiple concurrent Spark
> pipelines using {{spark-submit}} in Kubernetes cluster mode.
> The issue occurs when executors attempt to connect to the Spark driver. The
> driver hostname generated by {{DriverServiceFeatureStep}} contains the
> namespace and {{svc}} components:
> {{sample-app-driver-svc.default.svc}}
> Because this hostname contains fewer dots than the configured {{ndots}} value
> ({{{}ndots:5{}}}), the pod's DNS resolver first attempts to resolve the
> hostname using the configured search domains.
> This results in two failed DNS lookups ({{{}NXDOMAIN{}}}) before the driver
> service is successfully resolved.
> h2. Environment
> * *Apache Spark:* 3.5.2 and 4.1.1
> * *Deployment Mode:* Cluster mode on Kubernetes
> * *CoreDNS:* Standard Kubernetes configuration
> * *Pod DNS resolver:* {{ndots:5}}
> * *DNS search domains:*
> {{search default.svc.cluster.local svc.cluster.local cluster.local}}
> h2. Current Behaviour
> {{DriverServiceFeatureStep.scala}} currently constructs {{spark.driver.host}}
> as:
> {{val driverHostname =
> s"$resolvedServiceName.${kubernetesConf.namespace}.svc"}}
> For example:
> {{sample-app-driver-svc.default.svc}}
> The DNS resolver then performs the following lookups:
> {{Attempt 1:
> sample-app-driver-svc.default.svc.default.svc.cluster.local. → NXDOMAIN
> Attempt 2:
> sample-app-driver-svc.default.svc.svc.cluster.local. → NXDOMAIN
> Attempt 3:
> sample-app-driver-svc.default.svc.cluster.local. → NOERROR}}
> As a result, two unnecessary DNS lookups are performed before the driver
> service is resolved.
> h2. Impact
> We are running multiple Spark pipelines concurrently in our production
> Kubernetes environment.
> Each executor startup can generate these unnecessary failed DNS queries while
> attempting to connect to the driver. With a large number of concurrent
> pipelines and executors, this results in additional DNS traffic and can
> contribute to delays during executor-to-driver connection establishment.
> It also generates additional {{NXDOMAIN}} traffic to CoreDNS, which is a
> shared cluster-wide component.
> h2. Observed DNS Queries
> With the current hostname construction, CoreDNS receives:
>
> {{[INFO] "A IN sample-app-driver-svc.default.svc.default.svc.cluster.local."
> NXDOMAIN
> [INFO] "AAAA IN sample-app-driver-svc.default.svc.default.svc.cluster.local."
> NXDOMAIN
> [INFO] "A IN sample-app-driver-svc.default.svc.svc.cluster.local." NXDOMAIN
> [INFO] "AAAA IN sample-app-driver-svc.default.svc.svc.cluster.local." NXDOMAIN
> [INFO] "A IN sample-app-driver-svc.default.svc.cluster.local." NOERROR
> [INFO] "AAAA IN sample-app-driver-svc.default.svc.cluster.local." NOERROR}}
> h2. Proposed Solution
> One possible solution we have tested is to use only the Kubernetes Service
> name as {{{}spark.driver.host{}}}:
> {{sample-app-driver-svc}}
> The Kubernetes DNS search path would then resolve it directly as:
> {{sample-app-driver-svc.default.svc.cluster.local.}}
> This avoids the intermediate {{NXDOMAIN}} lookups and allows the service to
> be resolved on the first DNS attempt.
> The corresponding code change would be:
> *Current:*
> {{val driverHostname =
> s"$resolvedServiceName.${kubernetesConf.namespace}.svc"}}
> *Proposed:*
> {{val driverHostname = s"$resolvedServiceName"}}
> h2. Questions
> Before proposing a code change to {{{}DriverServiceFeatureStep{}}}, we would
> like to understand whether there is an existing Spark configuration or
> external property that can address this behaviour.
> Specifically:
> 1. Is there any Spark configuration/property that can be used to avoid these
> unnecessary DNS lookups without modifying {{{}DriverServiceFeatureStep{}}}?
> 2. Is there a specific reason why {{DriverServiceFeatureStep}} constructs
> {{spark.driver.host}} using {{.namespace.svc}} instead of relying on the
> standard Kubernetes DNS search path?
> 3. Would using only the Kubernetes Service name as {{spark.driver.host}} be a
> safe and supported approach for Kubernetes cluster mode?
> 4. If there is no existing configuration option, would changing the default
> hostname construction to use the Service name only be considered an
> appropriate fix?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
