sarutak commented on PR #59878: URL: https://github.com/apache/spark/pull/59878#issuecomment-6482355553
By the way, as I mentioned in #54004, there was a similar discussion for the Thrift server in SPARK-23078 (allow submitting Spark Thrift Server in cluster mode on K8s). It was resolved as Won't Do, and the PR (#20272) was closed without being merged, because K8s supports client mode inside a pod: you can define the server pod (e.g. a Deployment plus a Service) with a YAML manifest and run the server there directly in client mode. That approach also gives a stable Service name and a restart on failure, which the cluster mode driver pod (`restartPolicy: Never`) does not. The same seems to apply to the Spark Connect server, e.g. running `sbin/start-connect-server.sh --wait --master k8s://https://kubernetes.default.svc ...` in a Deployment, with a headless Service and `spark.kubernetes.driver.pod.name` as described in "Client Mode Networking" and "Client Mode Executor Pod Garbage Collection" in `docs/running-on-kubernetes.md`. YARN is different, because there is no lightweight way to run a client mode driver inside the cluster there. So, are there requirements for the Spark Connect server on K8s that this approach cannot meet and cluster mode can? If so, it would help to describe them in the PR description, since it would also explain why the Connect server is treated differently from the Thrift server here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
