yalindogusahin opened a new pull request, #753:
URL: https://github.com/apache/spark-kubernetes-operator/pull/753
### What changes were proposed in this pull request?
Remove the `spark.submit.deployMode: "cluster"` line from
`examples/spark-connect-server-with-spark-cluster.yaml` and replace it with a
comment clarifying that the operator runs the Connect server in the driver pod
it manages, so the application always runs in client deploy mode.
### Why are the changes needed?
The example declares cluster deploy mode for a `SparkApplication` whose main
class is `org.apache.spark.sql.connect.service.SparkConnectServer`, but the
setting is silently ignored: the operator launches the main class inside the
driver pod it creates, so the server effectively runs in client mode regardless
(verified on-cluster with operator 0.9.0 / chart 1.7.0, Spark 4.1.2 — runtime
behavior is identical with `deployMode` set to `cluster` and `client`).
Upstream `SparkSubmit` also explicitly rejects cluster deploy mode for the
Connect server ("Cluster deploy mode is not applicable to Spark Connect
server"). The example therefore implies a capability that doesn't exist, and a
user copying it silently gets client mode.
Note this is distinct from `examples/submit-spark-connect-server.sh`
(SPARK-51382), which submits via the standalone REST API on port 6066 and
bypasses the SparkSubmit-side check; that example is not touched.
See https://issues.apache.org/jira/browse/SPARK-58101.
### Does this PR introduce _any_ user-facing change?
No functional change — documentation/example fix only. The example now
reflects the actual runtime behavior.
### How was this patch tested?
Manually verified on a minikube cluster with spark-kubernetes-operator 0.9.0
(Helm chart 1.7.0) and Spark 4.1.2 attached to a standalone `SparkCluster`
master: applying the example with `deployMode` set to `cluster` and to `client`
produces identical behavior (Connect server runs in the operator-managed driver
pod in client mode in both cases).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Fable 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
