Github user mccheah commented on the issue: https://github.com/apache/spark/pull/21748 > Can you point in the fork where the submission client is create the headless service? (just to help me understand the internals) > Btw If we stick to this manual approach, the need for the manual headless service should be documented. @echarles we create the headless service in spark-submit as part of spark-submit only for cluster mode: https://github.com/apache/spark/blob/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala#L69 Note that we only invoke any of the feature steps and the entry point of `KubernetesClientApplication` if we run in cluster mode. If we run in client mode, we enter directly into the user's main class, or, the user is in a process that just created a Spark context from scratch with the right master URL (i.e. `new SparkContext(k8s://my-master:8443)`). If you wanted to create the headless service in client mode, you'd have to do it when instantiating the `KubernetesClusterSchedulerBackend` somewhere, probably before creating the `ExecutorPodsAllocator` so that you'd set `spark.driver.host` and `spark.driver.port` properly when telling the created executors where to find the driver via the `--driver-url` argument. I've deferred implementing this code path. The documentation for using a headless service is only suggested, but not mentioned as a hard requirement: https://github.com/apache/spark/pull/21748/files#diff-b5527f236b253e0d9f5db5164bdb43e9R131. I didn't put this as a hard requirement because I could imagine some users wanting to not specifically use a headless service for this; perhaps they want to use a full Service object and share that service object with ports to be exposed for other reachable endpoints their pod exposes.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org