sarutak commented on code in PR #54004:
URL: https://github.com/apache/spark/pull/54004#discussion_r2738577682
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -310,6 +310,9 @@ private[spark] class SparkSubmit extends Logging {
error("Cluster deploy mode is not applicable to Spark SQL shell.")
case (_, CLUSTER) if isThriftServer(args.mainClass) =>
error("Cluster deploy mode is not applicable to Spark Thrift server.")
+ case (YARN, CLUSTER) if isConnectServer(args.mainClass) =>
+ logInfo("SparkConnectServer is starting in cluster deploy mode." +
+ "Use `yarn application -kill` command or YARN client API to stop the
server.")
case (_, CLUSTER) if isConnectServer(args.mainClass) =>
error("Cluster deploy mode is not applicable to Spark Connect server.")
Review Comment:
I tried running SparkConnectServer on my K8s cluster and I noticed there is
no simple way to delete K8s resources for the server.
When SparkConnectServer runs in cluster deploy mode in K8s cluster, one
service resource and pods for Driver and Executors launch so users need to
delete each resource unless users set labels explicitly before launching
SparkConnectServer. I felt it's not so useful.
Another challenge is `LoggingPodStatusWatcherImpl` frequently (every 1s)
write log until SparkConnectServer stops. Users can change the frequency but it
affects other executor pods.
If someone needs to run SparkConnectServer in cluster deploy mode on K8s
I'll consider it but I'd like to focus on YARN for now. Or, do you need this
for K8s?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
