Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/21748
@echarles I don't think we should be special-casing Kubernetes here as
being any different from the other cluster managers. The main point of client
mode is that the driver is running locally and we make no assumptions about
network connectivity. Deploying notebooks on other cluster managers which have
network isolation between the driver and the executors would require the user
to manually configure their own connectivity too.
There's another problem in creating the headless service here: when you
create a service, you have to back that service with pods that have certain
labels; that is, a label selector. However, labels are not unique for a given
pod; multiple pods can have the same label key-value pairs. Now, when Spark
Submit has explicit control over the pod that is created as well as the
service, we make it such that the driver pod has a unique label corresponding
to an application id, and then the service can be ensured to be backed by only
that driver pod. If we don't control the creation of the driver pod, we can't
guarantee that whatever service we create would route to only the driver pod
running the application in question.
It is therefore unclear what the label selector would be for the headless
service we create automatically. You could allow the user to have some control
over the configuration of the headless service. For example, we could have a
setting indicating the label selector the headless service should be created
with. Or, the documentation can specifically require the driver pod to have a
set of labels which are strictly unique to that pod in that namespace. In both
cases, I would argue that we end up with the same complexity and mental burden,
if not more so, than having the user deploy a headless service that gives their
driver pod (and only that driver pod) a stable hostname.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
