zhengchenyu opened a new pull request, #57350:
URL: https://github.com/apache/spark/pull/57350
### What changes were proposed in this pull request?
Add config `spark.kubernetes.driver.ui.service.enabled` (default false).
When enabled:
* A new `DriverUIServiceFeatureStep` creates a dedicated ClusterIP Service
exposing the driver's Web UI port.
* After `SparkContext` initializes and the Web UI
binds,`K8sDriverUIServicePatcher` (invoked from
`KubernetesClusterSchedulerBackend.start()`) patches the Service's `targetPort`
to the actual bound port.
When `spark.ui.port=0`, the Service is initially built with the default UI
port as placeholder to satisfy Kubernetes port validation; the runtime patch
then rewrites `targetPort`.
Two other configs: `spark.kubernetes.driver.ui.service.type` (ClusterIP /
NodePort / LoadBalancer, default ClusterIP) and
`spark.kubernetes.driver.ui.service.name` (optional explicit name).
### Why are the changes needed?
In hostNetwork mode, `spark.ui.port=0` is needed to avoid UI port collisions
between drivers on the same host. Today the driver UI Service is created by
spark-operator, not by Spark itself, and spark-operator has no way to know the
actual UI port the driver ends up binding to.
Letting Spark create and manage the UI Service resolves this because Spark
knows the actual bound port and patches the Service accordingly.
### Does this PR introduce _any_ user-facing change?
Yes. Three new opt-in configs, default off.
### How was this patch tested?
* New `DriverUIServiceFeatureStepSuite`:
* Test on kubernetes cluster.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
