ericm-db commented on code in PR #56907:
URL: https://github.com/apache/spark/pull/56907#discussion_r3545750568
##########
docs/spark-connect-overview.md:
##########
@@ -277,6 +277,70 @@ The connection may also be programmatically created using
_SparkSession#builder_
</div>
</div>
+## Faster local iteration with a persistent Connect server
+
+When you develop or test locally with
+
+{% highlight python %}
+from pyspark.sql import SparkSession
+spark = SparkSession.builder.remote("local[*]").getOrCreate()
+{% endhighlight %}
+
+PySpark boots a fresh in-process Spark Connect server in **every** process.
Each
+`python script.py` run (or each forked test JVM) therefore re-pays the
one-time startup cost --
+JVM warmup, `SparkContext` construction, and Connect server boot -- which can
take a few seconds and
+makes a quick edit/run loop feel slow.
+
+There are two ways to amortize that cost across runs by reconnecting to a
long-lived local server.
+
+### Start a server yourself and connect to it
+
+Start one persistent local Spark Connect server and point every run at it:
+
+{% highlight bash %}
Review Comment:
Ah okay, changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
