j1wonpark opened a new pull request, #57242: URL: https://github.com/apache/spark/pull/57242
### What changes were proposed in this pull request? Add a new configuration `spark.kubernetes.driver.service.publishNotReadyAddresses` (boolean, default `false`). When enabled: - `DriverServiceFeatureStep` sets `spec.publishNotReadyAddresses: true` on the headless driver service, so DNS records for the driver pod are published even while the pod is not Ready. - The driver pod readiness wait before executor allocation (introduced in SPARK-32975) is skipped in `ExecutorPodsAllocator`, `StatefulSetPodsAllocator`, and `DeploymentPodsAllocator`, since the sole purpose of that wait — ensuring the headless service is resolvable by DNS before executors start — no longer depends on pod readiness. ### Why are the changes needed? Kubernetes publishes DNS records for a headless service only for Ready pods. Users commonly attach a readiness probe to the driver pod through `spark.kubernetes.driver.podTemplateFile` — a typical setup for long-running Spark Connect servers, where pod readiness gates traffic routing. When such a probe gates on a port that binds after `SparkContext` initialization completes (e.g. the Spark Connect gRPC port), executors launched during initialization cannot resolve the driver service and die with `UnknownHostException`. With static allocation, `maxNumExecutorFailures = max(3, 2 * spark.executor.instances)` is reached before initialization can complete, and the driver exits with code 11 (`EXCEED_MAX_EXECUTOR_FAILURES`) on every attempt — the application can never start. The existing mitigation from SPARK-32975 (`waitUntilReady` in the allocators, tunable via `spark.kubernetes.allocation.driver.readinessTimeout` per SPARK-49079) cannot help here: the wait runs inside `SparkContext` initialization, while such a probe can only pass after initialization completes, so the wait always burns the full timeout and the race resumes. The driver service is a single-pod discovery endpoint rather than a load-balancing service, and `publishNotReadyAddresses` is the standard Kubernetes pattern for headless discovery services. There is currently no first-class way to set it — the service spec is not customizable via pod templates, and the only workarounds are re-implementing the service in a custom feature step (via `spark.kubernetes.driver.pod.excludedFeatureSteps`) or cluster-level admission webhooks. This follows the precedent of exposing driver service spec fields as configurations (SPARK-39490: `ipFamilyPolicy`/`ipFamilies`). The configuration is opt-in (default `false`) because publishing not-ready addresses is not desirable for deployments that rely on endpoint readiness to gate traffic to the driver service (e.g. through a service mesh); flipping the default could be discussed separately. ### Does this PR introduce _any_ user-facing change? No change by default. A new opt-in configuration is added and documented in `running-on-kubernetes.md`. ### How was this patch tested? New unit tests: - `DriverServiceFeatureStepSuite`: the default leaves the field `false`; enabling the configuration sets it to `true`. - `ExecutorPodsAllocatorSuite`, `StatefulSetAllocatorSuite`, `DeploymentAllocatorSuite`: the driver pod readiness wait still runs by default, and enabling the configuration skips it (`waitUntilReady` is not invoked). Also verified the behavior end-to-end on a real Kubernetes cluster by applying `publishNotReadyAddresses: true` to the driver service through an admission webhook (equivalent to what this configuration does): executors resolved the driver service and registered while the driver pod was still NotReady behind a Spark Connect port readiness probe, and a static-allocation application that previously failed on every attempt with exit code 11 started successfully. ### Was this patch authored or co-authored using generative AI tooling? Yes. Co-authored with Claude Opus. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
