Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
> Note that we only invoke any of the feature steps and the entry point of
KubernetesClientApplication if we run in cluster mode. If we run in client
mode, we enter directly into the user's main class, or, the user is in a
process that just created a Spark context from scratch with the right master
URL (i.e. new SparkContext(k8s://my-master:8443)). If you wanted to create the
headless service in client mode, you'd have to do it when instantiating the
KubernetesClusterSchedulerBackend somewhere, probably before creating the
ExecutorPodsAllocator so that you'd set spark.driver.host and spark.driver.port
properly when telling the created executors where to find the driver via the
--driver-url argument.
For Out-Cluster we don't need any headless service, but for In-Cluster the
user has to ensure that the driver is accessible via a service (headless or
not) - I understand it may be preferable to align on spark other managers and
to have coherent behaviour for k8s in/out cluster, but I still believe that we
have to make life easy to users and devops.
The notebook case is self explaining : For Zeppelin you may have a single
pod (Web Server) that will create multiple client sessions (this is a common
supported scenario), Jupyter with Toree may be configured to support multiple
REPL (and if it not today, it will come on day). You also have the case of
current livy which is a single REST Server managing a pool of REPL.
With current PR, we push to all those 3rd party consumer the responsibility
to create services, but all the hard work to assign and maintain ports to avoid
collision on the single pod with multiple drivers.
What about having a property that would enable/disable that headless
service creation (could be disabled by default) - This is not yet the perfect
solution as the user still would have to pass the ports and ensure there is no
clash.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
