sunchao opened a new pull request, #57977: URL: https://github.com/apache/spark/pull/57977
### What changes were proposed in this pull request? - Add a DNS-free helper for recognizing IPv4 and IPv6 wildcard bind addresses, including `0.0.0.0`, `::`, `[::]`, and expanded IPv6 unspecified addresses. - Preserve the configured advertised driver host in both `SparkContext` and `BasicExecutorFeatureStep` when the driver bind address is a wildcard. - Continue using concrete driver Pod IPs and preserve existing hexadecimal IPv6 normalization. - Add regression coverage for default-enabled driver initialization, executor URLs, wildcard IPv4/IPv6 addresses, and concrete IPv4/IPv6 driver addresses. ### Why are the changes needed? [SPARK-58748](https://issues.apache.org/jira/browse/SPARK-58748) fixes Kubernetes client-mode and Spark Connect drivers that bind to all interfaces while advertising a separate routable driver address: ```properties spark.master=k8s://https://kubernetes.example:6443 spark.driver.bindAddress=0.0.0.0 spark.driver.host=10.0.0.42 spark.kubernetes.executor.useDriverPodIP=true ``` Currently, `SparkContext` overwrites the advertised driver host with `0.0.0.0`, and `BasicExecutorFeatureStep` independently constructs an unroutable executor endpoint such as `spark://[email protected]:7078`. Executors cannot register, so the application stalls. IPv6 wildcard bind addresses fail the same way. This affects Spark 4.1 and 4.2 when `spark.kubernetes.executor.useDriverPodIP=true` is explicitly enabled. Spark 4.3 and current master enable the option by default, so a Spark 4.3 backport is particularly important. ### Does this PR introduce _any_ user-facing change? Yes. Kubernetes drivers that bind to a wildcard address now continue advertising the configured routable `spark.driver.host`; executors can register successfully. Drivers that bind to a concrete Pod IP retain the existing direct-IP behavior and IPv6 normalization. ### How was this patch tested? ```bash build/sbt -Pkubernetes \ 'core/testOnly org.apache.spark.SparkContextSuite -- -z "SPARK-58748"' \ 'core/testOnly org.apache.spark.util.UtilsSuite -- -z "SPARK-58748"' \ 'kubernetes/testOnly org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStepSuite' \ 'core/scalastyle' \ 'core/Test/scalastyle' \ 'kubernetes/scalastyle' \ 'kubernetes/Test/scalastyle' ``` Results: 1 targeted `SparkContextSuite` test, 1 targeted `UtilsSuite` test, all 39 `BasicExecutorFeatureStepSuite` tests, and all four Scala style checks passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
