shazebkhan1 commented on PR #58283: URL: https://github.com/apache/spark/pull/58283#issuecomment-5476360272
Hi @dongjoon-hyun, We wanted to share our experience with SPARK-53944 and also report a related issue that we discovered during testing. **SPARK-53944 — Backported and validated on Spark 3.5.2** We cherry-picked the changes from PRs #52650, #52923, and #52954 (`Config.scala`, `BasicExecutorFeatureStep.scala`, and `SparkContext.scala`) onto a clean Spark 3.5.2 (`v3.5.2`) base. We also included the SPARK-57351 default-to-true change from PR #56412. We rebuilt both `spark-core` and `spark-kubernetes` JARs, deployed the patched artifacts, and confirmed the following: * Executor pods now connect directly to the driver pod IP instead of using the driver Service hostname. * The NXDOMAIN queries for the driver Service name are no longer present in the CoreDNS logs. The fix works as expected. Thank you for the work on this. **New issue — API server NXDOMAIN queries from the driver pod** After resolving the executor-to-driver DNS issue, we observed another set of NXDOMAIN queries in CoreDNS. These queries originate from the driver pod itself when it communicates with the Kubernetes API server. For example: ```text NXDOMAIN kubernetes.default.svc.default.svc.cluster.local. NXDOMAIN kubernetes.default.svc.svc.cluster.local. NOERROR kubernetes.default.svc.cluster.local. ``` This pattern occurs repeatedly during API server communication. The behavior appears to be related to the `ndots:5` resolver configuration. Since `kubernetes.default.svc` contains only two dots, the resolver first appends the search domains from `/etc/resolv.conf` before trying the name as an absolute domain. This results in unnecessary NXDOMAIN queries before the correct name is resolved. **Root cause** `SparkKubernetesClientFactory.createKubernetesClient()` first calls Fabric8's `autoConfigure()`. When running inside Kubernetes, Fabric8 correctly detects the in-cluster environment using `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT` and configures the API server using the in-cluster IP, for example: ```text https://10.96.0.1:443 ``` However, Spark then unconditionally overrides this configuration using: ```scala .withMasterUrl(master) ``` where `master` defaults to: ```scala val KUBERNETES_MASTER_INTERNAL_URL = "https://kubernetes.default.svc"; ``` This effectively overrides Fabric8's in-cluster configuration and forces the client to use the Kubernetes Service hostname instead of the API server IP. We also observed that Fabric8's `IpAddressMatcher` internally calls `InetAddress.getByName()` on this hostname, resulting in additional DNS lookups. We have confirmed that this behavior is present in Spark 3.5.x, `branch-4.1`, and the current master (5.0-SNAPSHOT). This issue is separate from SPARK-53944: SPARK-53944 addresses executor-to-driver connectivity, whereas this issue affects driver-to-Kubernetes API server connectivity. **Proposed fix** In `SparkKubernetesClientFactory.scala`, we propose skipping `.withMasterUrl(master)` when both of the following conditions are true: 1. `KUBERNETES_SERVICE_HOST` is present, indicating that the driver is running inside Kubernetes. 2. `master` is the default `KUBERNETES_MASTER_INTERNAL_URL`, meaning that the user has not configured a custom master URL. In this case, Fabric8's `autoConfigure()` has already configured the correct in-cluster API server IP, so there is no need to override it. The proposed change is: ```scala val baseConfig = new ConfigBuilder(autoConfigure(kubeContext.orNull)) .withApiVersion("v1") // When running in-cluster with the default master URL, Fabric8 autoConfigure() has already // configured the API server using KUBERNETES_SERVICE_HOST / KUBERNETES_SERVICE_PORT. // Overriding it with the hostname causes ndots:5 search-domain expansion and unnecessary // DNS lookups on API server calls. Skip the override so the in-cluster IP is used directly. val configWithMaster = if (sys.env.contains("KUBERNETES_SERVICE_HOST") && master == KUBERNETES_MASTER_INTERNAL_URL) { logInfo( "Running in-cluster with default master URL; using API server IP from " + "KUBERNETES_SERVICE_HOST to avoid unnecessary DNS lookups." ) baseConfig } else { baseConfig.withMasterUrl(master) } val config = configWithMaster .withRequestTimeout(clientType.requestTimeout(sparkConf)) // ... rest unchanged ``` **Compatibility** We believe this change should be safe for existing use cases: * **External-cluster deployments:** `KUBERNETES_SERVICE_HOST` is not present, so `withMasterUrl(master)` is still called. * **Custom `spark.kubernetes.driver.master`:** The master value differs from `KUBERNETES_MASTER_INTERNAL_URL`, so `withMasterUrl(master)` is still called. * **Non-Kubernetes deployments:** There is no change in behavior. Would the team be open to a PR for this change? We would be happy to contribute it against `master`. Thanks in advance for your feedback. Best regards, Shazeb Khan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
