HyukjinKwon commented on code in PR #49107:
URL: https://github.com/apache/spark/pull/49107#discussion_r1890010605
##########
docs/app-dev-spark-connect.md:
##########
@@ -58,6 +58,59 @@ to the client via the blue box as part of the Spark Connect
API. The client uses
alongside PySpark or the Spark Scala client, making it easy for Spark client
applications to work
with the custom logic/library.
+## Spark API Mode: Spark Client and Spark Classic
+
+Spark provides API mode (`spark.api.mode`) configuration that can seamlessly
start Spark Classic
+applications to use Spark Connect. Based on `spark.api.mode` configuration,
the application either
+be executed in Spark Classic or Spark Connect mode. See the examples below:
+
+{% highlight python %}
+from pyspark.sql import SparkSession
+
+SparkSession.builder.config("spark.api.mode",
"connect").master("...").getOrCreate()
+{% endhighlight %}
+
+
+This configuration can be also effectively set for both Scala Spark
application and PySpark
+applications via Spark submission:
+
+{% highlight bash %}
+spark-submit --master "..." --conf spark.api.mode=connect
+{% endhighlight %}
+
+In addition, Spark Connect provides convient special cases for you to locally
test. `spark.remote`
+can be set as `local[...]` and `local-cluster[...]`. In this case, it creates
a locally running
+Spark Connect server, and provide users the Spark Connect session. This is
similar when
+`--conf spark.api.mode=connect` and `--master ...` are set together. However
`spark.remote`
+and `--remote` only supports `local*` whereas `--conf spark.api.mode=connect`
and `--master ...`
+allow other Spark cluster URLs such as `spark://` for better compatibility.
Review Comment:
@hvanhovell this part is slightly different from what we discussed offline.
My logic here is to restrict it first instead of allowing it: `--remote` and
`spark.remote` will support only `local*` as it was.
##########
docs/app-dev-spark-connect.md:
##########
@@ -58,6 +58,34 @@ to the client via the blue box as part of the Spark Connect
API. The client uses
alongside PySpark or the Spark Scala client, making it easy for Spark client
applications to work
with the custom logic/library.
+## Spark API Mode: Spark Client and Spark Classic
+
+Spark provides the API mode, `spark.api.mode` configuration, enabling Spark
Classic applications
+to seamlessly switch to Spark Connect. Depending on the value of
`spark.api.mode`, the application
+can run in either Spark Classic or Spark Connect mode. Here is an example:
+
+{% highlight python %}
+from pyspark.sql import SparkSession
+
+SparkSession.builder.config("spark.api.mode",
"connect").master("...").getOrCreate()
+{% endhighlight %}
+
+You can also apply this configuration to both Scala and PySpark applications
when submitting yours:
+
+{% highlight bash %}
+spark-submit --master "..." --conf spark.api.mode=connect
+{% endhighlight %}
+
+Additionally, Spark Connect offers convenient options for local testing. By
setting `spark.remote`
+to `local[...]` or `local-cluster[...]`, you can start a local Spark Connect
server and access a Spark
+Connect session.
+
+This is similar to using `--conf spark.api.mode=connect` with `--master ...`.
However, note that
+`spark.remote` and `--remote` are limited to `local*` values, while `--conf
spark.api.mode=connect`
+with `--master ...` supports additional cluster URLs, such as spark://, for
broader compatibility with
+Spark Classic.
Review Comment:
@hvanhovell this part is slightly different from what we discussed offline.
My logic here is to restrict it first instead of allowing it: `--remote` and
`spark.remote` will support only `local*` as it was.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
