Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
We discussed this offline. After some experimentation, we concluded that
it's not actually straightforward to set up the headless service in the
Kubernetes scheduler code in client mode, which would be where we'd have to put
this code. The problem is that before the scheduler starts up, the driver needs
to bind to the host given by `spark.driver.host`. But if that hostname is tied
to a headless service and the headless service does not exist, the hostname
bind will fail.
There's a few ways to work around this, but all of them seem risky to put
in a first pass at this patch.
Additionally, it's not clear if the scheduler code should be opinionated
about configuring network connectivity for the driver. Client mode is just a
definition of the driver running in a local process - Spark currently doesn't
make any assumptions about what that local process is, whether it's in a
Kubernetes pod or not. Contrast this with cluster mode where the driver should
know it's running in a pod as submitted by `KubernetesClientApplication`, and
`KubernetesClientApplication` can know to set up its headless service.
Therefore we are not going to have the driver set up the headless service
in this patch. If we decide later on that creating the headless service is the
right thing to do, we can introduce that functionality in a separate patch.
I'm going to update the docs and hope to merge this by the end of the day
on Monday, July 23. Let us know if there are any additional concerns. Thanks
@liyinan926 @echarles.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
