This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new d24be719dd6 [SPARK-42656][CONNECT][FOLLOWUP] Improve the script to
start spark-connect server
d24be719dd6 is described below
commit d24be719dd6a4b5f3548cdb8cd7872cd7084dd82
Author: Zhen Li <[email protected]>
AuthorDate: Mon Mar 6 22:05:16 2023 -0400
[SPARK-42656][CONNECT][FOLLOWUP] Improve the script to start spark-connect
server
### What changes were proposed in this pull request?
Make the server to start with Hive dependency and make the script to build
with the correct scala version set in pom files.
Also allow the server to take any extra spark configurations.
### Why are the changes needed?
Adding hive dependency to allow some tests of write ops easier.
Allow the server to start with the same Scala version set in the project.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually tested.
Closes #40303 from zhenlineo/improve-scripts.
Authored-by: Zhen Li <[email protected]>
Signed-off-by: Herman van Hovell <[email protected]>
(cherry picked from commit d4818df19c73120edd9878907e87abaed20d4871)
Signed-off-by: Herman van Hovell <[email protected]>
---
connector/connect/bin/spark-connect | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/connector/connect/bin/spark-connect
b/connector/connect/bin/spark-connect
index 008209c8440..2f2ce7df08c 100755
--- a/connector/connect/bin/spark-connect
+++ b/connector/connect/bin/spark-connect
@@ -22,11 +22,12 @@ FWDIR="$(cd "`dirname "$0"`"/../../..; pwd)"
cd "$FWDIR"
export SPARK_HOME=$FWDIR
-# Build the jars needed for spark submit and spark connect
-build/sbt package
-
SCALA_BINARY_VER=`grep "scala.binary.version" "${SPARK_HOME}/pom.xml" | head
-n1 | awk -F '[<>]' '{print $3}'`
+SCALA_ARG=$(if [ "${SCALA_BINARY_VER}" == "2.13" ]; then echo "-Pscala-2.13";
else echo ""; fi)
+
+# Build the jars needed for spark submit and spark connect
+build/sbt "${SCALA_ARG}" -Phive package
CONNECT_JAR=`ls
"${SPARK_HOME}"/connector/connect/server/target/scala-"${SCALA_BINARY_VER}"/spark-connect-assembly*.jar
| paste -sd ',' -`
-exec "${SPARK_HOME}"/bin/spark-submit --class
org.apache.spark.sql.connect.SimpleSparkConnectService "$CONNECT_JAR"
\ No newline at end of file
+exec "${SPARK_HOME}"/bin/spark-submit "$@" --class
org.apache.spark.sql.connect.SimpleSparkConnectService "$CONNECT_JAR"
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
