Github user echarles commented on the issue:
https://github.com/apache/spark/pull/21748
Thx @mccheah and @liyinan926. It now works with an headless service using
`spark.driver.host=spark-driver-service` and
`spark.kubernetes.driver.pod.name=spark-pod`
Two more questions:
1. I still have an issue with Out-Cluster: Executors are also killed on
start. Is there something I should configure to make it work?
2. With the PR I had developer on the fork (long time ago), I didn't have
to create by myself the headless service. I have looked at the difference
between fork and apache, but don't see where it comes from. In short, as a
user, it would be better if I hadn't to create by myself the headless service,
especially if we consider an notebook environment where a WEB Server running in
a pod would launch multiple REPL (in client mode), I see practical issue to
manage the assignment of the exposed ports (you can not assign a port which is
already assigned). Thoughts?
For info, the specs of the headless service which is needed to run
In-Cluster.
```
apiVersion: v1
kind: Service
metadata:
name: spark-driver-service
spec:
clusterIP: None
ports:
- port: 7077
name: spark-driver-port
- port: 10000
name: spark-driver-blockmanager-block
selector:
run: spark-pod
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
