pan3793 commented on code in PR #54004:
URL: https://github.com/apache/spark/pull/54004#discussion_r2739369496
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -310,6 +310,9 @@ private[spark] class SparkSubmit extends Logging {
error("Cluster deploy mode is not applicable to Spark SQL shell.")
case (_, CLUSTER) if isThriftServer(args.mainClass) =>
error("Cluster deploy mode is not applicable to Spark Thrift server.")
+ case (YARN, CLUSTER) if isConnectServer(args.mainClass) =>
+ logInfo("SparkConnectServer is starting in cluster deploy mode." +
+ "Use `yarn application -kill` command or YARN client API to stop the
server.")
case (_, CLUSTER) if isConnectServer(args.mainClass) =>
error("Cluster deploy mode is not applicable to Spark Connect server.")
Review Comment:
for how to delete the resource, does this work? it's similar to `yarn
application -kill <application id>`
```
spark-submit --kill [submission ID] --master [spark://...]
```
> Another challenge is LoggingPodStatusWatcherImpl frequently (every 1s)
write log until SparkConnectServer stops.
do you mean `spark-submit` process or driver pod? for the former,
`spark.kubernetes.submission.waitAppCompletion=false` might help, for the
latter, it does not seems an issue to me.
> I'd like to focus on YARN for now.
fine to me for the current status of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
