This is an automated email from the ASF dual-hosted git repository. hvanhovell pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push: new d24be719dd6 [SPARK-42656][CONNECT][FOLLOWUP] Improve the script to start spark-connect server d24be719dd6 is described below commit d24be719dd6a4b5f3548cdb8cd7872cd7084dd82 Author: Zhen Li <zhenli...@users.noreply.github.com> AuthorDate: Mon Mar 6 22:05:16 2023 -0400 [SPARK-42656][CONNECT][FOLLOWUP] Improve the script to start spark-connect server ### What changes were proposed in this pull request? Make the server to start with Hive dependency and make the script to build with the correct scala version set in pom files. Also allow the server to take any extra spark configurations. ### Why are the changes needed? Adding hive dependency to allow some tests of write ops easier. Allow the server to start with the same Scala version set in the project. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually tested. Closes #40303 from zhenlineo/improve-scripts. Authored-by: Zhen Li <zhenli...@users.noreply.github.com> Signed-off-by: Herman van Hovell <her...@databricks.com> (cherry picked from commit d4818df19c73120edd9878907e87abaed20d4871) Signed-off-by: Herman van Hovell <her...@databricks.com> --- connector/connect/bin/spark-connect | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/connector/connect/bin/spark-connect b/connector/connect/bin/spark-connect index 008209c8440..2f2ce7df08c 100755 --- a/connector/connect/bin/spark-connect +++ b/connector/connect/bin/spark-connect @@ -22,11 +22,12 @@ FWDIR="$(cd "`dirname "$0"`"/../../..; pwd)" cd "$FWDIR" export SPARK_HOME=$FWDIR -# Build the jars needed for spark submit and spark connect -build/sbt package - SCALA_BINARY_VER=`grep "scala.binary.version" "${SPARK_HOME}/pom.xml" | head -n1 | awk -F '[<>]' '{print $3}'` +SCALA_ARG=$(if [ "${SCALA_BINARY_VER}" == "2.13" ]; then echo "-Pscala-2.13"; else echo ""; fi) + +# Build the jars needed for spark submit and spark connect +build/sbt "${SCALA_ARG}" -Phive package CONNECT_JAR=`ls "${SPARK_HOME}"/connector/connect/server/target/scala-"${SCALA_BINARY_VER}"/spark-connect-assembly*.jar | paste -sd ',' -` -exec "${SPARK_HOME}"/bin/spark-submit --class org.apache.spark.sql.connect.SimpleSparkConnectService "$CONNECT_JAR" \ No newline at end of file +exec "${SPARK_HOME}"/bin/spark-submit "$@" --class org.apache.spark.sql.connect.SimpleSparkConnectService "$CONNECT_JAR" \ No newline at end of file --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org