pan3793 opened a new pull request, #6642: URL: https://github.com/apache/kyuubi/pull/6642
This is a quick and dirty PR that implements the Spark Connect (gRPC-based) protocol, which allows you to use `pyspark` or `spark-shell` with `remote sc://<host>:<port>` mode to connect Kyuubi and run Spark queries. DISCLAIMER: this patch is not ready for review, and I know it breaks the existing functionalities, which will be fixed later. For explorers who want to try this early version: ### Source Code under development ``` git clone https://github.com/pan3793/kyuubi.git -b kyuubi-next kyuubi-next ``` ### Requirements - `JAVA_HOME` points to Java 17 - `SPARK_HOME` points to `/path/of/spark-4.0.0-preview1-bin-hadoop3` ### Run #### Run within IDEA - Run `build/mvn clean install -DskipTests` to build the project and produce the Spark engine jar - Run `kyuubi-server/src/main/scala/org/apache/kyuubi/server/KyuubiServer.scala` using IDEA You can set `SPARK_HOME`, `KYUUBI_CONF_DIR` in the Run/Debug Configuration Dialog <img width="912" alt="idea_run_debug_configurations_dialog" src="https://github.com/user-attachments/assets/67544b5a-ea3c-4974-8fd6-b6085e9bca46";> #### Run within Terminal ``` build/dist cd dist bin/kyuubi run --conf kyuubi.frontend.grpc.bind.port=10999 ``` The gRPC service listens 10999 by default. ### Connect to Kyuubi Connect Spark Connect Scala client(use the latest `spark-shell` as example) ``` git clone g...@github.com:apache/spark.git cd spark build/sbt package -Phive SPARK_PREPEND_CLASSES=1 bin/spark-shell --remote sc://H27212-MAC-01.local:10999 --user_id chengpan --user_name chengpan ``` PySpark Connect client (Requires: Python >=3.9) ``` pip install pyspark-connect==4.0.0.dev1 pyspark --remote sc://H27212-MAC-01.local:10999 --user_id chengpan --user_name chengpan ``` Run examples ``` Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 4.0.0-SNAPSHOT /_/ Type in expressions to have them evaluated. Spark session available as 'spark'. scala> spark.sql("select 1").show() +---+ | 1| +---+ | 1| +---+ ``` --- # Checklist 📝 <!--- Go over all the following points, and put an `x` in all the boxes that apply. --> <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: notifications-unsubscr...@kyuubi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: notifications-unsubscr...@kyuubi.apache.org For additional commands, e-mail: notifications-h...@kyuubi.apache.org
