nchammas commented on code in PR #56907:
URL: https://github.com/apache/spark/pull/56907#discussion_r3544983656
##########
docs/spark-connect-overview.md:
##########
@@ -277,6 +277,70 @@ The connection may also be programmatically created using
_SparkSession#builder_
</div>
</div>
+## Faster local iteration with a persistent Connect server
+
+When you develop or test locally with
+
+{% highlight python %}
+from pyspark.sql import SparkSession
+spark = SparkSession.builder.remote("local[*]").getOrCreate()
+{% endhighlight %}
+
+PySpark boots a fresh in-process Spark Connect server in **every** process.
Each
+`python script.py` run (or each forked test JVM) therefore re-pays the
one-time startup cost --
+JVM warmup, `SparkContext` construction, and Connect server boot -- which can
take a few seconds and
+makes a quick edit/run loop feel slow.
+
+There are two ways to amortize that cost across runs by reconnecting to a
long-lived local server.
+
+### Start a server yourself and connect to it
+
+Start one persistent local Spark Connect server and point every run at it:
+
+{% highlight bash %}
Review Comment:
Nit: These should all be Markdown code fences (i.e. ```bash) instead of
Liquid tags. They will still render correctly, and the hash symbols will not be
mistaken for headings (like they are here) when editing the raw Markdown.
##########
docs/spark-connect-overview.md:
##########
@@ -277,6 +277,70 @@ The connection may also be programmatically created using
_SparkSession#builder_
</div>
</div>
+## Faster local iteration with a persistent Connect server
+
+When you develop or test locally with
+
+{% highlight python %}
+from pyspark.sql import SparkSession
+spark = SparkSession.builder.remote("local[*]").getOrCreate()
+{% endhighlight %}
+
+PySpark boots a fresh in-process Spark Connect server in **every** process.
Each
+`python script.py` run (or each forked test JVM) therefore re-pays the
one-time startup cost --
+JVM warmup, `SparkContext` construction, and Connect server boot -- which can
take a few seconds and
+makes a quick edit/run loop feel slow.
+
+There are two ways to amortize that cost across runs by reconnecting to a
long-lived local server.
+
+### Start a server yourself and connect to it
+
+Start one persistent local Spark Connect server and point every run at it:
+
+{% highlight bash %}
+# Start once; it stays up across runs.
+$SPARK_HOME/sbin/start-connect-server.sh --master "local[*]"
+
+# Every run reconnects instead of booting a new server.
+python -c 'from pyspark.sql import SparkSession;
SparkSession.builder.remote("sc://localhost:15002").getOrCreate()'
+
+# Stop it when you are done.
+$SPARK_HOME/sbin/stop-connect-server.sh
+{% endhighlight %}
+
+### Let PySpark manage the server (opt-in)
+
+If you would rather keep your code as
`SparkSession.builder.remote("local[*]").getOrCreate()` and not
+manage a server by hand, enable the opt-in reuse path. The first run starts a
**detached** local
+Connect server and records it in a discovery file; later runs reconnect to it
in a fraction of a
+second:
+
+{% highlight bash %}
+export SPARK_LOCAL_CONNECT_REUSE=1 # or
.config("spark.local.connect.reuse", "true")
+python script.py # 1st run: starts a persistent server (cold start, once)
+python script.py # 2nd+ run: reconnects to it (sub-second)
+{% endhighlight %}
+
+This is **off by default**; nothing changes unless you opt in. A few details:
+
+- Each run connects as its own Connect session, so session-local state -- temp
views, runtime SQL
+ configurations, and (with artifact isolation, which stays on) session
artifacts -- is fresh on
+ every run and never leaks between runs. State backed by the shared
`SparkContext` (the persistent
+ catalog/warehouse, global temp views, and cached datasets) *is* shared
across runs, so namespace
+ per-run databases or clear that state yourself if your runs must be fully
isolated.
+- The server listens on port `15002` by default and authenticates with a token
written, together
+ with the host, port, pid and Spark version, to `~/.spark/connect-local.json`
(mode `0600`).
Review Comment:
As far as I can tell, this is the first use of `~/.spark/` in the project.
There are some aspects to discuss here since there has been some
[pushback](https://dotfiles-matter.click) against storing app data under `~/`,
even though it is very common.
On Linux, there is the [XDG Base Directory
Specification](https://specifications.freedesktop.org/basedir/latest/). On
macOS, there is `~/Library/` and the [various directories
underneath](https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPFileSystem/Articles/WhereToPutFiles.html).
Windows probably has its own setup, too.
Python has the [platformdirs](https://github.com/tox-dev/platformdirs)
library for taking care of these platform-specific differences. Perhaps
Scala/Java also has something similar.
We should consider whether we want to try to follow these platform-specific
conventions, even though our lives would be easier in the short term to just
put things under `~/.spark/`. Another idea is to use a specific path under
`SPARK_HOME`. I think it would be a small win if we could eliminate the need
for a new config or env var that's specific to this file's location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
