Yalin Dogu Sahin created SPARK-58101:
----------------------------------------
Summary: Misleading spark.submit.deployMode=cluster in
spark-connect-server-with-spark-cluster.yaml example
Key: SPARK-58101
URL: https://issues.apache.org/jira/browse/SPARK-58101
Project: Spark
Issue Type: Improvement
Components: Connect, Kubernetes
Affects Versions: kubernetes-operator-0.9.0
Reporter: Yalin Dogu Sahin
[examples/spark-connect-server-with-spark-cluster.yaml|https://github.com/apache/spark-kubernetes-operator/blob/main/examples/spark-connect-server-with-spark-cluster.yaml]
declares
{{spark.submit.deployMode=cluster}} for a {{SparkApplication}} whose main class
is {{org.apache.spark.sql.connect.service.SparkConnectServer}}, attached to a
standalone {{SparkCluster}} master.
However, the operator runs the main class inside the driver pod it manages, so
the application effectively runs in *client mode regardless of this setting*
(verified on-cluster with operator 0.9.0 / chart 1.7.0,
Spark 4.1.2: runtime behavior is identical with {{deployMode}} set to
{{cluster}} and {{client}}). Upstream {{SparkSubmit}} also explicitly rejects
cluster deploy mode for the Connect server ({{"Cluster deploy mode
is not applicable to Spark Connect server"}}), so the example implies a
capability that doesn't exist and a user copying it will silently get client
mode.
Note this is distinct from {{examples/submit-spark-connect-server.sh}}
(SPARK-51382), which submits via the standalone REST API on port 6066 and
bypasses the SparkSubmit-side check, so that path is not affected.
Proposals (any of):
# Remove the {{spark.submit.deployMode}} line from the example, or
# Add a comment clarifying that the operator runs the Connect server in client
mode regardless, or
# Have the operator warn or reject when cluster deploy mode is declared for a
Connect server.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
