hiboyang commented on PR #59878: URL: https://github.com/apache/spark/pull/59878#issuecomment-6503198707
> By the way, as I mentioned in #54004, there was a similar discussion for the Thrift server in [SPARK-23078](https://issues.apache.org/jira/browse/SPARK-23078) (allow submitting Spark Thrift Server in cluster mode on K8s). It was resolved as Won't Do, and the PR (#20272) was closed without being merged, because K8s supports client mode inside a pod: you can define the server pod (e.g. a Deployment plus a Service) with a YAML manifest and run the server there directly in client mode. That approach also gives a stable Service name and a restart on failure, which the cluster mode driver pod (`restartPolicy: Never`) does not. > > The same seems to apply to the Spark Connect server, e.g. running `sbin/start-connect-server.sh --wait --master k8s://https://kubernetes.default.svc ...` in a Deployment, with a headless Service and `spark.kubernetes.driver.pod.name` as described in "Client Mode Networking" and "Client Mode Executor Pod Garbage Collection" in `docs/running-on-kubernetes.md`. YARN is different, because there is no lightweight way to run a client mode driver inside the cluster there. > > So, are there requirements for the Spark Connect server on K8s that this approach cannot meet and cluster mode can? If so, it would help to describe them in the PR description, since it would also explain why the Connect server is treated differently from the Thrift server here. Hi @sarutak, yes, thanks for the discussion and the context! Agreed that running start-connect-server.sh in client mode inside a Deployment works well for a long-running server. This PR is a small change so users who already launch jobs with spark-submit in K8s cluster mode can start SparkConnectServer the same way (same pod templates and configs, no separate Deployment/Service manifests), as YARN already allows. I'll add this to the PR description :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
