peter-toth commented on code in PR #59878:
URL: https://github.com/apache/spark/pull/59878#discussion_r4411112277
##########
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala:
##########
@@ -464,6 +464,38 @@ class SparkSubmitSuite
conf.get("spark.kubernetes.driver.container.image") should be ("bar")
}
+ test("handles Spark Connect server in k8s cluster mode") {
Review Comment:
**Finding 3.** Nit: new tests usually carry the JIRA ID.
```suggestion
test("SPARK-60566: handles Spark Connect server in k8s cluster mode") {
```
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -313,8 +313,15 @@ private[spark] class SparkSubmit extends Logging {
case (YARN, CLUSTER) if isConnectServer(args.mainClass) =>
logInfo("SparkConnectServer is starting in cluster deploy mode. " +
"Use `yarn application -kill` command or YARN client API to stop the
server.")
+ case (KUBERNETES, CLUSTER) if isConnectServer(args.mainClass) =>
+ logInfo("SparkConnectServer is starting in cluster deploy mode. " +
+ "Use `spark-submit --kill <namespace>:<driver-pod-name>` or delete
the driver pod " +
+ "to stop the server. Consider setting " +
Review Comment:
**Finding 2.** `spark-submit --kill` needs the K8s master as well. Without
`--master`, `args.master` falls back to `local[*]`
(`SparkSubmitArguments.scala:47`). Then `SparkSubmitUtils.getSubmitOperations`
finds no operation that supports it, and throws "No external
SparkSubmitOperations clients found for master url: 'local[*]'"
(`SparkSubmit.scala:1259`). So the command only works when `spark.master` is
set in `spark-defaults.conf`. The K8s docs pass `--master` too
(`docs/running-on-kubernetes.md:858`). The [earlier
comment](https://github.com/apache/spark/pull/59878#discussion_r4405981651) had
`--master k8s://...` in its text, but not in its example.
```suggestion
"Use `spark-submit --master <k8s-master-url> --kill
<namespace>:<driver-pod-name>` " +
"or delete the driver pod to stop the server. Consider setting " +
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
