nchammas commented on code in PR #59613: URL: https://github.com/apache/spark/pull/59613#discussion_r4457068182
########## docs/spark-connect-setup.md: ########## @@ -0,0 +1,529 @@ +--- +layout: global +title: Setting up Spark Connect +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Spark Connect ships with every Spark release since 3.4 and supports PySpark and Scala +applications. For what Spark Connect is and how it differs from classic Spark, see the +[Spark Connect Overview](spark-connect-overview.html). + +* This will become a table of contents (this text will be scraped). +{:toc} + +## Starting a Spark Connect server + +First, download Spark from the +[Download Apache Spark](https://spark.apache.org/downloads.html) page. Choose the +latest release in the release drop down at the top of the page. Then choose your package type, typically +“Pre-built for Apache Hadoop 3.5 and later”, and click the link to download. + +Now extract the Spark package you just downloaded on your computer, for example: + +```bash +tar -xvf spark-{{site.SPARK_VERSION_SHORT}}-bin-hadoop3.tgz +``` + +In a terminal window, go to the `spark` folder in the location where you extracted +Spark before and run the `start-connect-server.sh` script to start Spark server with +Spark Connect. If you already have Spark installed and `SPARK_HOME` defined, you can use that too. + +```bash +./sbin/start-connect-server.sh + +# alternately +"$SPARK_HOME/sbin/start-connect-server.sh" +``` + +Alternatively, `./bin/spark-connect-shell` starts an interactive Scala shell with the Connect server hosted inside the shell process itself. + +The script accepts the same options as `spark-submit`, so `--master`, `--jars`, `--packages` +and `--conf` work as they do for any other Spark application. Everything that the server +reads at startup, such as `spark.sql.extensions`, `spark.jars`, `spark.jars.packages` and the +JARs that catalogs load, is set here. Clients can only change runtime configuration of their own +session; that includes defining a catalog (`spark.sql.catalog.<name>`) whose JAR is already on the +server. + +By default the server listens on port 15002 on all interfaces and runs as a background daemon +(`./sbin/stop-connect-server.sh` stops it). Pass `--wait` as the first argument to keep it in +the foreground instead, for example as a container entrypoint or under a process supervisor. With +`--wait`, the script sets `SPARK_NO_DAEMONIZE=1`, writes no PID file and sends the logs to standard +output. `spark.connect.grpc.binding.address` restricts the listening interface and +`spark.connect.grpc.binding.port` changes the port: + +```bash +./sbin/start-connect-server.sh --wait \ + --conf spark.connect.grpc.binding.address=127.0.0.1 \ + --conf spark.connect.grpc.binding.port=15002 +``` + +Spark does not retry another port when the configured one is taken +(`spark.connect.grpc.port.maxRetries` defaults to 0), so a port conflict fails the server with a +`java.net.BindException` instead of silently moving it. A daemonized server logs to `logs/` under +the Spark folder (or `SPARK_LOG_DIR`), and the Spark UI of the server, at port 4040 by default, has a +**Connect** tab listing sessions and requests. + +## Interactive analysis with Spark Connect +<div class="codetabs"> + +<div data-lang="python" markdown="1"> +When creating a Spark session, you can specify that you want to use Spark Connect +and there are a few ways to do that outlined as follows. + +If you do not use one of the mechanisms outlined here, your Spark session will +work just like before, without leveraging Spark Connect. + +### Set SPARK_REMOTE environment variable + +If you set the `SPARK_REMOTE` environment variable on the client machine where your +Spark client application is running and create a new Spark Session as in the following +example, the session will be a Spark Connect session. With this approach, there is no +code change needed to start using Spark Connect. + +In a terminal window, set the `SPARK_REMOTE` environment variable to point to the +local Spark server you started previously on your computer: + +```bash +export SPARK_REMOTE="sc://localhost" +``` + +And start the Spark shell as usual: + +```bash +./bin/pyspark +``` + +The PySpark shell is now connected to Spark using Spark Connect as indicated in the welcome message: + +```python +Client connected to the Spark Connect server at localhost +``` + +### Specify Spark Connect when creating Spark session + +You can also specify that you want to use Spark Connect explicitly when you +create a Spark session. + +For example, you can launch the PySpark shell with Spark Connect as +illustrated here. + +To launch the PySpark shell with Spark Connect, simply include the `remote` +parameter and specify the location of your Spark server. We are using `localhost` +in this example to connect to the local Spark server we started previously: + +```bash +./bin/pyspark --remote "sc://localhost" +``` + +And you will notice that the PySpark shell welcome message tells you that +you have connected to Spark using Spark Connect: + +```python +Client connected to the Spark Connect server at localhost +``` + +You can also check the Spark session type. If it includes `.connect.` you +are using Spark Connect as shown in this example: + +```python +SparkSession available as 'spark'. +>>> type(spark) +<class 'pyspark.sql.connect.session.SparkSession'> +``` + +Now you can run PySpark code in the shell to see Spark Connect in action: + +```python +>>> columns = ["id", "name"] +>>> data = [(1,"Sarah"), (2,"Maria")] +>>> df = spark.createDataFrame(data).toDF(*columns) +>>> df.show() ++---+-----+ +| id| name| ++---+-----+ +| 1|Sarah| +| 2|Maria| ++---+-----+ +``` + +</div> + +<div data-lang="scala" markdown="1"> +For the Scala shell, we use an Ammonite-based REPL. Otherwise, very similar with PySpark shell. + +```bash +./bin/spark-shell --remote "sc://localhost" +``` + +A greeting message will appear when the REPL successfully initializes: +```bash +Welcome to + ____ __ + / __/__ ___ _____/ /__ + _\ \/ _ \/ _ `/ __/ '_/ + /___/ .__/\_,_/_/ /_/\_\ version {{site.SPARK_VERSION}} + /_/ + +Type in expressions to have them evaluated. +Spark session available as 'spark'. +``` + +By default, the REPL will attempt to connect to a local Spark Server. +Run the following Scala code in the shell to see Spark Connect in action: + +```scala +@ spark.range(10).count +res0: Long = 10L +``` + +</div> +</div> + +## Connection strings + +By default, the shells and the `remote()` builder attempt to connect to a local Spark Server on +port 15002. The connection, however, may be configured in several ways as described in this +configuration [reference](https://github.com/apache/spark/blob/master/sql/connect/docs/client-connection-string.md). +A connection string has the form `sc://host:port/;param1=value;param2=value`: the scheme is +always `sc://`, the path component must stay empty because gRPC uses the HTTP/2 path for its own +method names, and parameters such as `token`, `use_ssl`, `user_id` and `user_agent` follow a `/;` +separator. Setting `token` turns on TLS for any host except `localhost`; to use TLS with +`localhost`, set `use_ssl=true` as well. + +In the Python client, a parameter value cannot contain a literal `/`, because its parser reads +everything from the first `/` as the path: + +```python +SparkSession.builder.remote("sc://localhost:15002/;user_agent=myapp/1").getOrCreate() +# [INVALID_CONNECT_URL] Invalid URL for Spark Connect: +# The path component '/;user_agent=myapp/1' must be empty. +``` + +Write the `/` as `%2F` instead: `user_agent=myapp%2F1` is accepted and read as `myapp/1`. + +### Set SPARK_REMOTE environment variable + +The SPARK_REMOTE environment variable can be set on the client machine to customize the client-server +connection that is initialized at REPL startup. + +```bash +export SPARK_REMOTE="sc://myhost.com:443/;token=ABCDEFG" +./bin/spark-shell +``` + +### Configure programmatically with a connection string + +The connection may also be programmatically created using _SparkSession#builder_ as in this example: + +<div class="codetabs"> + +<div data-lang="python" markdown="1"> +```python +from pyspark.sql import SparkSession +spark = SparkSession.builder.remote("sc://localhost:443/;token=ABCDEFG").getOrCreate() +``` +</div> + +<div data-lang="scala" markdown="1"> +```scala +@ import org.apache.spark.sql.SparkSession +@ val spark = SparkSession.builder.remote("sc://localhost:443/;token=ABCDEFG").getOrCreate() +``` +</div> Review Comment: Earlier we tell users: > Setting `token` turns on TLS for any host except `localhost`; to use TLS with `localhost`, set `use_ssl=true` as well. However in these examples we are using `localhost` without `use_ssl`. Let's fix the examples to follow the earlier guidance. Either drop the token, use a different host, or keep `localhost` and add `use_ssl`. ########## docs/spark-connect-setup.md: ########## @@ -0,0 +1,529 @@ +--- +layout: global +title: Setting up Spark Connect +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Spark Connect ships with every Spark release since 3.4 and supports PySpark and Scala +applications. For what Spark Connect is and how it differs from classic Spark, see the +[Spark Connect Overview](spark-connect-overview.html). + +* This will become a table of contents (this text will be scraped). +{:toc} + +## Starting a Spark Connect server + +First, download Spark from the +[Download Apache Spark](https://spark.apache.org/downloads.html) page. Choose the +latest release in the release drop down at the top of the page. Then choose your package type, typically +“Pre-built for Apache Hadoop 3.5 and later”, and click the link to download. + +Now extract the Spark package you just downloaded on your computer, for example: + +```bash +tar -xvf spark-{{site.SPARK_VERSION_SHORT}}-bin-hadoop3.tgz +``` + +In a terminal window, go to the `spark` folder in the location where you extracted +Spark before and run the `start-connect-server.sh` script to start Spark server with +Spark Connect. If you already have Spark installed and `SPARK_HOME` defined, you can use that too. + +```bash +./sbin/start-connect-server.sh + +# alternately +"$SPARK_HOME/sbin/start-connect-server.sh" +``` + +Alternatively, `./bin/spark-connect-shell` starts an interactive Scala shell with the Connect server hosted inside the shell process itself. + +The script accepts the same options as `spark-submit`, so `--master`, `--jars`, `--packages` +and `--conf` work as they do for any other Spark application. Everything that the server +reads at startup, such as `spark.sql.extensions`, `spark.jars`, `spark.jars.packages` and the +JARs that catalogs load, is set here. Clients can only change runtime configuration of their own +session; that includes defining a catalog (`spark.sql.catalog.<name>`) whose JAR is already on the +server. + +By default the server listens on port 15002 on all interfaces and runs as a background daemon +(`./sbin/stop-connect-server.sh` stops it). Pass `--wait` as the first argument to keep it in +the foreground instead, for example as a container entrypoint or under a process supervisor. With +`--wait`, the script sets `SPARK_NO_DAEMONIZE=1`, writes no PID file and sends the logs to standard +output. `spark.connect.grpc.binding.address` restricts the listening interface and +`spark.connect.grpc.binding.port` changes the port: + +```bash +./sbin/start-connect-server.sh --wait \ + --conf spark.connect.grpc.binding.address=127.0.0.1 \ + --conf spark.connect.grpc.binding.port=15002 +``` + +Spark does not retry another port when the configured one is taken +(`spark.connect.grpc.port.maxRetries` defaults to 0), so a port conflict fails the server with a +`java.net.BindException` instead of silently moving it. A daemonized server logs to `logs/` under +the Spark folder (or `SPARK_LOG_DIR`), and the Spark UI of the server, at port 4040 by default, has a +**Connect** tab listing sessions and requests. + +## Interactive analysis with Spark Connect +<div class="codetabs"> + +<div data-lang="python" markdown="1"> +When creating a Spark session, you can specify that you want to use Spark Connect +and there are a few ways to do that outlined as follows. + +If you do not use one of the mechanisms outlined here, your Spark session will +work just like before, without leveraging Spark Connect. + +### Set SPARK_REMOTE environment variable + +If you set the `SPARK_REMOTE` environment variable on the client machine where your +Spark client application is running and create a new Spark Session as in the following +example, the session will be a Spark Connect session. With this approach, there is no +code change needed to start using Spark Connect. + +In a terminal window, set the `SPARK_REMOTE` environment variable to point to the +local Spark server you started previously on your computer: + +```bash +export SPARK_REMOTE="sc://localhost" +``` + +And start the Spark shell as usual: + +```bash +./bin/pyspark +``` + +The PySpark shell is now connected to Spark using Spark Connect as indicated in the welcome message: + +```python +Client connected to the Spark Connect server at localhost +``` + +### Specify Spark Connect when creating Spark session + +You can also specify that you want to use Spark Connect explicitly when you +create a Spark session. + +For example, you can launch the PySpark shell with Spark Connect as +illustrated here. + +To launch the PySpark shell with Spark Connect, simply include the `remote` +parameter and specify the location of your Spark server. We are using `localhost` +in this example to connect to the local Spark server we started previously: + +```bash +./bin/pyspark --remote "sc://localhost" +``` + +And you will notice that the PySpark shell welcome message tells you that +you have connected to Spark using Spark Connect: + +```python +Client connected to the Spark Connect server at localhost +``` + +You can also check the Spark session type. If it includes `.connect.` you +are using Spark Connect as shown in this example: + +```python +SparkSession available as 'spark'. +>>> type(spark) +<class 'pyspark.sql.connect.session.SparkSession'> +``` + +Now you can run PySpark code in the shell to see Spark Connect in action: + +```python +>>> columns = ["id", "name"] +>>> data = [(1,"Sarah"), (2,"Maria")] +>>> df = spark.createDataFrame(data).toDF(*columns) +>>> df.show() ++---+-----+ +| id| name| ++---+-----+ +| 1|Sarah| +| 2|Maria| ++---+-----+ +``` + +</div> + +<div data-lang="scala" markdown="1"> +For the Scala shell, we use an Ammonite-based REPL. Otherwise, very similar with PySpark shell. + +```bash +./bin/spark-shell --remote "sc://localhost" +``` + +A greeting message will appear when the REPL successfully initializes: +```bash +Welcome to + ____ __ + / __/__ ___ _____/ /__ + _\ \/ _ \/ _ `/ __/ '_/ + /___/ .__/\_,_/_/ /_/\_\ version {{site.SPARK_VERSION}} + /_/ + +Type in expressions to have them evaluated. +Spark session available as 'spark'. +``` + +By default, the REPL will attempt to connect to a local Spark Server. +Run the following Scala code in the shell to see Spark Connect in action: + +```scala +@ spark.range(10).count +res0: Long = 10L +``` + +</div> +</div> + +## Connection strings + +By default, the shells and the `remote()` builder attempt to connect to a local Spark Server on +port 15002. The connection, however, may be configured in several ways as described in this +configuration [reference](https://github.com/apache/spark/blob/master/sql/connect/docs/client-connection-string.md). +A connection string has the form `sc://host:port/;param1=value;param2=value`: the scheme is +always `sc://`, the path component must stay empty because gRPC uses the HTTP/2 path for its own +method names, and parameters such as `token`, `use_ssl`, `user_id` and `user_agent` follow a `/;` +separator. Setting `token` turns on TLS for any host except `localhost`; to use TLS with +`localhost`, set `use_ssl=true` as well. + +In the Python client, a parameter value cannot contain a literal `/`, because its parser reads +everything from the first `/` as the path: + +```python +SparkSession.builder.remote("sc://localhost:15002/;user_agent=myapp/1").getOrCreate() +# [INVALID_CONNECT_URL] Invalid URL for Spark Connect: +# The path component '/;user_agent=myapp/1' must be empty. +``` + +Write the `/` as `%2F` instead: `user_agent=myapp%2F1` is accepted and read as `myapp/1`. + +### Set SPARK_REMOTE environment variable + +The SPARK_REMOTE environment variable can be set on the client machine to customize the client-server +connection that is initialized at REPL startup. + +```bash +export SPARK_REMOTE="sc://myhost.com:443/;token=ABCDEFG" +./bin/spark-shell +``` + +### Configure programmatically with a connection string + +The connection may also be programmatically created using _SparkSession#builder_ as in this example: + +<div class="codetabs"> + +<div data-lang="python" markdown="1"> +```python +from pyspark.sql import SparkSession +spark = SparkSession.builder.remote("sc://localhost:443/;token=ABCDEFG").getOrCreate() +``` +</div> + +<div data-lang="scala" markdown="1"> +```scala +@ import org.apache.spark.sql.SparkSession +@ val spark = SparkSession.builder.remote("sc://localhost:443/;token=ABCDEFG").getOrCreate() +``` +</div> +</div> + +## Faster local iteration with a persistent Connect server +When you develop or test locally with + +```python +from pyspark.sql import SparkSession +spark = SparkSession.builder.remote("local[*]").getOrCreate() +``` + +PySpark boots a fresh in-process Spark Connect server that lives only as long as that Python +process. Every `python script.py` invocation (or every new test worker process) therefore re-pays +the one-time startup cost -- JVM warmup, `SparkContext` construction, and Connect server boot -- +which can take a few seconds and makes a quick edit/run loop feel slow. + +The in-process server binds port 15002 by default and the client dials `sc://localhost`, +so if another Spark Connect server is already listening on that port (a container, or a server +left running from an earlier session) the start fails with +`java.net.BindException: Failed to bind to address 0.0.0.0/0.0.0.0:15002`. Moving the port with +`spark.connect.grpc.binding.port` does not help here, because the client would still dial +15002 and reach the other server; stop the other server or connect to it by URL instead. + +To amortize that cost across runs, start one persistent local Spark Connect server and point +every run at it: + +```bash +# Start once; it stays up across runs. (--master is optional; it defaults to local[*].) +$SPARK_HOME/sbin/start-connect-server.sh --master "local[*]" + +# Every run reconnects instead of booting a new server. +python -c 'from pyspark.sql import SparkSession; SparkSession.builder.remote("sc://localhost:15002").getOrCreate()' + +# Stop it when you are done. +$SPARK_HOME/sbin/stop-connect-server.sh +``` + +Alternatively, on POSIX systems PySpark can manage this persistent server for you. With +`SPARK_LOCAL_CONNECT_REUSE=1` set (or `spark.local.connect.reuse=true` on the builder), +`SparkSession.builder.remote("local[*]").getOrCreate()` starts a persistent server through +`sbin/start-connect-server.sh` on the first run and reconnects to it on later runs, so scripts +keep the plain `local[*]` URL: + +```bash +export SPARK_LOCAL_CONNECT_REUSE=1 + +# The first run starts the server; later runs reconnect to it. +python -c 'from pyspark.sql import SparkSession; SparkSession.builder.remote("local[*]").getOrCreate()' + +# Stop the managed server when you are done. +python -m pyspark.sql.connect.local_server --stop +``` + +The managed server is an ordinary `spark-daemon.sh` daemon, but it runs with a per-user pid +directory and ident string so it cannot collide with a server you started by hand -- which also +means a plain `sbin/stop-connect-server.sh` does not find it. The `--stop` command signals the +recorded server and cleans up the discovery file; killing the server's pid directly also works, +and the next run notices the dead server and starts a fresh one. + +This managed-server workflow is experimental. The `--stop` command and the discovery file +location and format may change in a future release, for example if local server management is +folded into a unified `spark connect` CLI. + +The connection details (host, port, auth token, pid, Spark version) are recorded in a discovery +file in a private per-user directory; set `SPARK_LOCAL_CONNECT_DISCOVERY` to override its +location. A run reconnects only to a server whose Spark version matches. After upgrading Spark, +the server from the previous version cannot be reused and the next run fails with an error asking +you to stop it; run the `--stop` command above and rerun to start a fresh server. + +Each run connects as its own Connect session, so session-local state -- temp views, runtime SQL +configurations, and session artifacts -- is fresh on every run and never leaks between runs. State +backed by the shared `SparkContext` (the persistent catalog/warehouse, global temp views, and +cached datasets) *is* shared across runs, so namespace per-run databases or clear that state +yourself if your runs must be fully isolated. + +## Spark Connect in standalone applications +<div class="codetabs"> + + +<div data-lang="python" markdown="1"> + +First, install PySpark with `pip install pyspark-client=={{site.SPARK_VERSION_SHORT}}` or if building a packaged PySpark application/library, +add it your setup.py file as: +```python +install_requires=[ +'pyspark-client=={{site.SPARK_VERSION_SHORT}}' +] +``` + +When writing your own code, include the `remote` function with a reference to +your Spark server when you create a Spark session, as in this example: + +```python +from pyspark.sql import SparkSession +spark = SparkSession.builder.remote("sc://localhost").getOrCreate() +``` + + +For illustration purposes, we’ll create a simple Spark Connect application, SimpleApp.py: +```python +"""SimpleApp.py""" +from pyspark.sql import SparkSession + +logFile = "YOUR_SPARK_HOME/README.md" # a path the server can read +spark = SparkSession.builder.remote("sc://localhost").appName("SimpleApp").getOrCreate() +logData = spark.read.text(logFile).cache() + +numAs = logData.filter(logData.value.contains('a')).count() +numBs = logData.filter(logData.value.contains('b')).count() + +print("Lines with a: %i, lines with b: %i" % (numAs, numBs)) + +spark.stop() +``` + +This program just counts the number of lines containing ‘a’ and the number containing ‘b’ in a text file. +Note that you’ll need to replace YOUR_SPARK_HOME with the location where Spark is installed. + +We can run this application with the regular Python interpreter as follows: +```python +# Use the Python interpreter to run your application +$ python SimpleApp.py +... +Lines with a: 72, lines with b: 39 +``` +</div> + + +<div data-lang="scala" markdown="1"> +To use Spark Connect as part of a Scala application/project, we first need to include the right dependencies. +Using the `sbt` build system as an example, we add the following dependencies to the `build.sbt` file: +```sbt +libraryDependencies += "org.apache.spark" %% "spark-connect-client-jvm" % "{{site.SPARK_VERSION_SHORT}}" +``` + +Depend on `spark-connect-client-jvm` only, and on JDK 17 and later start the JVM with +`--add-opens=java.base/java.nio=ALL-UNNAMED`. + +When writing your own code, include the `remote` function with a reference to +your Spark server when you create a Spark session, as in this example: + +```scala +import org.apache.spark.sql.SparkSession +val spark = SparkSession.builder().remote("sc://localhost").getOrCreate() +``` + + +**Note**: Operations that reference User Defined Code such as UDFs, filter, map, etc require a +[ClassFinder](https://github.com/apache/spark/blob/master/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ClassFinder.scala) +to be registered to pickup and upload any required classfiles. Also, any JAR dependencies must be uploaded to the server using `SparkSession#AddArtifact`. + +Example: +```scala +import org.apache.spark.sql.connect.client.REPLClassDirMonitor +// Register a ClassFinder to monitor and upload the classfiles from the build output. +val classFinder = new REPLClassDirMonitor(<ABSOLUTE_PATH_TO_BUILD_OUTPUT_DIR>) +spark.registerClassFinder(classFinder) + +// Upload JAR dependencies +spark.addArtifact(<ABSOLUTE_PATH_JAR_DEP>) +``` +Here, `ABSOLUTE_PATH_TO_BUILD_OUTPUT_DIR` is the output directory where the build system writes classfiles into +and `ABSOLUTE_PATH_JAR_DEP` is the location of the JAR on the local file system. + +The `REPLClassDirMonitor` is a provided implementation of `ClassFinder` that monitors a specific directory but +one may implement their own class extending `ClassFinder` for customized search and monitoring. + +</div> +</div> + +For extending Spark Connect with custom server-side functionality, see +[Extending Spark with Spark Server Libraries](spark-connect-server-libs.html). + +## Switching between Spark Connect and Spark Classic + +Spark provides the API mode, `spark.api.mode` configuration, enabling Spark Classic applications +to switch to Spark Connect. Depending on the value of `spark.api.mode`, the application +can run in either Spark Classic or Spark Connect mode. Here is an example: + +```python +from pyspark.sql import SparkSession + +SparkSession.builder.config("spark.api.mode", "connect").master("...").getOrCreate() +``` + +You can also apply this configuration to both Scala and PySpark applications when submitting yours: + +```bash +spark-submit --master "..." --conf spark.api.mode=connect +``` + +Additionally, Spark Connect offers convenient options for local testing. By setting `spark.remote` +to `local[...]` or `local-cluster[...]`, you can start a local Spark Connect server and access a Spark +Connect session. + +This is similar to using `--conf spark.api.mode=connect` with `--master ...`. However, a `local*` +value of `spark.remote` or `--remote` starts a local server and an `sc://` value connects to an +existing server, while `--conf spark.api.mode=connect` with `--master ...` supports additional cluster URLs, such as spark://, for broader compatibility with +Spark Classic. Review Comment: This is a bit difficult to parse. I would restructure this to be a list that shows the multiple ways you can connect to a Spark Connect server, noting when you would use each. For example: - `sc://...` connects to a server that is already running. - `local*` starts a server inside your process. - `--master X` plus `spark.api.mode=connect` starts a dedicated server on cluster X. ########## docs/spark-connect-server-libs.md: ########## @@ -47,72 +36,20 @@ Client applications connect to Spark using the Spark Connect API, which is essen DataFrame API and fully declarative. <p style="text-align: center;"> - <img src="img/extending-spark-connect.png" title="Figure 1: Architecture" alt="Extending Spark -Connect Diagram"/> + <img + src="img/extending-spark-connect-labelled.png" + alt="Extending Spark Connect Diagram" + /> + <figcaption style="text-align: center;">Figure 1: Architecture</figcaption> </p> + Spark Server Libraries extend Spark. They typically provide additional server-side logic integrated with Spark, which is exposed to client applications as part of the Spark Connect API, using Spark Connect extension points. For example, the _Spark Server Library_ consists of custom -service-side logic (as indicated by the blue box labeled _Custom Library Plugin_), which is exposed -to the client via the blue box as part of the Spark Connect API. The client uses this API, e.g., -alongside PySpark or the Spark Scala client, making it easy for Spark client applications to work -with the custom logic/library. - -## Spark API Mode: Spark Client and Spark Classic - -Spark provides the API mode, `spark.api.mode` configuration, enabling Spark Classic applications -to seamlessly switch to Spark Connect. Depending on the value of `spark.api.mode`, the application -can run in either Spark Classic or Spark Connect mode. Here is an example: - -{% highlight python %} -from pyspark.sql import SparkSession - -SparkSession.builder.config("spark.api.mode", "connect").master("...").getOrCreate() -{% endhighlight %} - -You can also apply this configuration to both Scala and PySpark applications when submitting yours: - -{% highlight bash %} -spark-submit --master "..." --conf spark.api.mode=connect -{% endhighlight %} - -Additionally, Spark Connect offers convenient options for local testing. By setting `spark.remote` -to `local[...]` or `local-cluster[...]`, you can start a local Spark Connect server and access a Spark -Connect session. - -This is similar to using `--conf spark.api.mode=connect` with `--master ...`. However, note that -`spark.remote` and `--remote` are limited to `local*` values, while `--conf spark.api.mode=connect` -with `--master ...` supports additional cluster URLs, such as spark://, for broader compatibility with -Spark Classic. - -## Spark Client Applications - -Spark Client Applications are the _regular Spark applications_ that Spark users develop today, e.g., -ETL pipelines, data preparation, or model training or inference. These are typically built using -Sparks declarative DataFrame and DataSet APIs. With Spark Connect, the core behaviour remains the -same, but there are a few differences: -* Lower-level, non-declarative APIs (RDDs) can no longer be directly used from Spark Client -applications. Alternatives for missing RDD functionality are provided as part of the higher-level -DataFrame API. -* Client applications no longer have direct access to the Spark driver JVM; they are fully -separated from the server. - -Client applications based on Spark Connect can be submitted in the same way as any previous job. -In addition, Spark Client Applications based on Spark Connect have several benefits compared to -classic Spark applications using earlier Spark versions (3.4 and below): -* _Upgradability_: Upgrading to new Spark Server versions is seamless, as the Spark Connect API -abstracts any changes/improvements on the server side. Client- and server APIs are cleanly -separated. -* _Simplicity_: The number of APIs exposed to the user is reduced from 3 to 2. The Spark Connect API -is fully declarative and consequently easy to learn for new users familiar with SQL. -* _Stability_: When using Spark Connect, the client applications no longer run on the Spark driver -and, therefore don’t cause and are not affected by any instability on the server. -* _Remote connectivity_: The decoupled architecture allows remote connectivity to Spark beyond SQL -and JDBC: any application can now interactively use Spark “as a service”. -* _Backwards compatibility_: The Spark Connect API is code-compatible with earlier Spark versions, -except for the usage of RDDs, for which a list of alternative APIs is provided in Spark Connect. - -## Spark Server Libraries +service-side logic (the blue box labeled _Plugin_, with its _App Logic_), which is exposed to Review Comment: Typo: Server-side ########## docs/spark-connect-overview.md: ########## @@ -70,387 +71,50 @@ that developers need to be aware of when using Spark Connect: 2. By design, the Spark Connect protocol uses Sparks logical Review Comment: Nit: "Spark's logical plans" ########## docs/spark-connect-server-libs.md: ########## @@ -47,72 +36,20 @@ Client applications connect to Spark using the Spark Connect API, which is essen DataFrame API and fully declarative. <p style="text-align: center;"> - <img src="img/extending-spark-connect.png" title="Figure 1: Architecture" alt="Extending Spark -Connect Diagram"/> + <img + src="img/extending-spark-connect-labelled.png" + alt="Extending Spark Connect Diagram" + /> + <figcaption style="text-align: center;">Figure 1: Architecture</figcaption> Review Comment: `<figcaption>` is only valid inside `<figure>`, no? ########## docs/spark-connect-server-libs.md: ########## @@ -137,16 +72,19 @@ A Spark Server Library consists of the following components, illustrated in Fig. 4. The client package that exposes the Spark Server Library application logic to the Spark Client Application, alongside PySpark or the Scala Spark Client. <p style="text-align: center;"> - <img src="img/extending-spark-connect-labelled.png" title="Figure 2: Labelled Architecture" alt="Extending Spark -Connect Diagram - Labelled Steps"/> -</p> + <img + src="img/extending-spark-connect-labelled.png" + alt="Extending Spark Connect Diagram - Labelled Steps" + /> + <figcaption style="text-align: center;">Figure 2: Labelled Architecture</figcaption> Review Comment: Whoops, I was looking at the wrong page. My mistake. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
