sunchao commented on code in PR #6949: URL: https://github.com/apache/datafusion-comet/pull/6949#discussion_r4265720336
########## docs/source/user-guide/latest/kubernetes.md: ########## @@ -61,21 +61,26 @@ REVISION: 1 TEST SUITE: None ``` -Create example Spark application file `spark-pi.yaml` +Create an example Spark SQL application file named `comet-sql.yaml`. The query uses a generated +range, so it does not require an external data source. Its `EXPLAIN` output also makes it possible +to verify that Comet is executing the query. ```bash apiVersion: sparkoperator.k8s.io/v1beta2 kind: SparkApplication metadata: - name: spark-pi + name: comet-sql namespace: default spec: type: Scala mode: cluster image: apache/datafusion-comet imagePullPolicy: IfNotPresent - mainClass: org.apache.spark.examples.SparkPi - mainApplicationFile: local:///opt/spark/examples/jars/spark-examples_2.13-4.1.3.jar + mainClass: org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver Review Comment: [P2] Use a main class that supports cluster deployment. The manifest still sets `mode: cluster`, which the Spark Operator forwards as `--deploy-mode cluster`. Spark explicitly identifies `SparkSQLCLIDriver` as the SQL shell and rejects this combination, including when `-e` supplies a noninteractive query. Following the example therefore fails before creating the driver instead of reaching `COMPLETED` and producing the documented Comet plan/result. Could this use a small cluster-compatible SQL/DataFrame application instead? Evidence: Reproduced with stock Spark 4.1.3: `spark-submit --master k8s://https://127.0.0.1:65534 --deploy-mode cluster --class org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver local:///opt/spark/jars/spark-hive-thriftserver_2.13-4.1.3.jar -e 'EXPLAIN SELECT sum(id * id) FROM range(1000000); SELECT sum(id * id) FROM range(1000000);'` exits 1 with `Cluster deploy mode is not applicable to Spark SQL shell.` before contacting Kubernetes. Spark v4.1.3 `core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:309-310` implements the rejection, and lines 1197-1198 identify this exact main class. The SQL succeeds separately in client mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
