Chenyu Zheng created SPARK-58203:
------------------------------------
Summary: Support configuring spark-managed UI service to work with
spark.ui.port=0
Key: SPARK-58203
URL: https://issues.apache.org/jira/browse/SPARK-58203
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 4.3.0
Reporter: Chenyu Zheng
In hostNetwork mode, setting {{spark.ui.port=0}} (kernel-picked random port) is
required to avoid UI port collisions when multiple drivers share the host's
port space.
Today the driver UI Service is created by spark-operator, not by Spark itself.
spark-operator has no way to know the actual UI port that the driver's UI
server ends up binding to.
Proposal: provide configs to let Spark create and manage the UI Service itself.
* Add config {{spark.kubernetes.driver.ui.service.enabled}} (default false).
When enabled, Spark creates a dedicated ClusterIP UI Service.
* After the driver's Web UI binds, Spark patches the Service's {{targetPort}}
to the actual bound port.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
