This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7c0e66c4794a [SPARK-50182][EXAMPLE] Add `submit-sql.sh` REST API
example
7c0e66c4794a is described below
commit 7c0e66c4794ae000bc64fe50bb029c1a7b4c250f
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Oct 30 22:11:24 2024 -0700
[SPARK-50182][EXAMPLE] Add `submit-sql.sh` REST API example
### What changes were proposed in this pull request?
This PR aims to provide `submit-sql.sh` example via REST API.
### Why are the changes needed?
To provide the REST API feature more clearly via a simple SQL example.
Especially, this PR contributes a local-cluster mode example which is
useful for light and small SQL Spark jobs.
### Does this PR introduce _any_ user-facing change?
No, this is a new example.
### How was this patch tested?
Manual review.
Start Spark Cluster and Submit the job via `submit-sql.sh` script.
```
$ SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true" sbin/start-master.sh
$ sbin/start-worker.sh spark://$(hostname):7077
$ ./examples/src/main/scripts/submit-sql.sh
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #48714 from dongjoon-hyun/SPARK-50182.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
examples/src/main/scripts/submit-sql.sh | 39 +++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/examples/src/main/scripts/submit-sql.sh
b/examples/src/main/scripts/submit-sql.sh
new file mode 100755
index 000000000000..4b61106514a8
--- /dev/null
+++ b/examples/src/main/scripts/submit-sql.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Usage: Run SQL statements
+#
+# submit-sql.sh <spark_master_hostname>
+#
+
+SPARK_MASTER=${1:-localhost}
+
+curl -XPOST http://$SPARK_MASTER:6066/v1/submissions/create \
+--data '{
+ "appResource": "",
+ "sparkProperties": {
+ "spark.master": "local[2]",
+ "spark.driver.cores": "2",
+ "spark.driver.memory": "2g"
+ },
+ "clientSparkVersion": "",
+ "mainClass": "org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver",
+ "action": "CreateSubmissionRequest",
+ "appArgs": [ "-e", "SHOW DATABASES; SELECT * FROM RANGE(10)" ]
+}'
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
