ericm-db opened a new pull request, #56907:
URL: https://github.com/apache/spark/pull/56907
### What changes were proposed in this pull request?
This PR adds an **opt-in** way to make local Spark Connect development and
testing start much
faster, by reconnecting to a persistent local Spark Connect server instead
of booting a fresh
in-process server in every process.
Today `SparkSession.builder.remote("local[*]").getOrCreate()` starts a
brand-new in-process Spark
Connect server in each process (see `SparkSession._start_connect_server`).
Every `python script.py`
run (and every forked test JVM) therefore re-pays the one-time startup cost
-- JVM warmup,
`SparkContext` construction, and Connect server boot.
When the new opt-in is enabled (`SPARK_LOCAL_CONNECT_REUSE=1`, or
`.config("spark.local.connect.reuse", "true")`), a local-mode remote session
instead:
1. probes a discovery file (`~/.spark/connect-local.json`, overridable via
`SPARK_LOCAL_CONNECT_DISCOVERY`) and reconnects to the recorded server if
it is still alive,
the port is open, and the Spark version matches; otherwise
2. launches a **detached** local Connect server
(`pyspark/sql/connect/local_server.py`), waits until
it is accepting connections, and records it in the discovery file so the
next process reconnects.
The user's code is unchanged -- still
`SparkSession.builder.remote("local[*]").getOrCreate()`. The
first run pays the cold start once; later runs reconnect in a fraction of a
second.
Details:
- **Default behavior is unchanged.** Nothing happens unless the opt-in is
set; the existing
in-process path is untouched.
- **No protocol or Scala changes** -- this is entirely in PySpark plus one
new error condition.
- **Auth:** a stable token is minted and written to the discovery file (mode
`0600`) together with
the host, port, pid, and Spark version. The client authenticates with it
via
`SPARK_CONNECT_AUTHENTICATE_TOKEN`.
- **Isolation:** each run connects as its own Connect session, so
session-local state (temp views,
runtime SQL configs, and -- with artifact isolation, which stays on --
session artifacts) is fresh
per run. Only state backed by the shared `SparkContext` (persistent
catalog, global temp views,
cached datasets) is shared across runs, as documented.
- **Lifecycle:** the server self-terminates after
`spark.local.connect.server.idleTimeout` seconds idle (default 3600; `0`
disables), and can be
stopped explicitly via `SparkSession._stop_local_connect_server()`.
- **Staleness:** a dead pid, closed port, or version mismatch makes reuse
fall back to starting a
fresh server and rewriting the discovery file.
Open points for reviewers (called out for discussion):
- The opt-in name (`SPARK_LOCAL_CONNECT_REUSE` /
`spark.local.connect.reuse`).
- Whether auto-spawning a detached server from a library call is acceptable,
vs. requiring an
explicit start command.
- The idle-timeout default and mechanism.
### Why are the changes needed?
Creating a local Spark session for a quick edit/run loop "consistently takes
almost 3 seconds ...
it makes the whole development experience feel slow." That cost is
one-time-per-process and does not
amortize across separate runs, so the only way to make a repeated dev/test
loop fast is to keep a
warm server alive and reconnect to it. This change makes that
reconnect-or-start behavior available
behind a single opt-in, without changing user code or default behavior.
### Does this PR introduce _any_ user-facing change?
Yes, but only when the new opt-in is explicitly enabled. With
`SPARK_LOCAL_CONNECT_REUSE=1` (or
`spark.local.connect.reuse=true`),
`SparkSession.builder.remote("local[*]").getOrCreate()` starts a
persistent local Connect server on the first run and reconnects to it on
later runs, instead of
booting a fresh in-process server every time. With the opt-in unset (the
default), behavior is
unchanged. A new documentation section ("Faster local iteration with a
persistent Connect server")
describes the feature.
### How was this patch tested?
Added `python/pyspark/sql/tests/connect/test_connect_local_server.py`:
- unit tests for the discovery-file read/path logic and the reuse decision
(version mismatch, dead
pid, alive-and-listening, safe no-op stop);
- an end-to-end test that starts a real detached local Connect server,
verifies a second call
reconnects to the same server (same pid, no respawn), runs queries over
two independent client
connections, and asserts session-local state (a temp view) does not leak
across them, then stops
the server and confirms the process exits and the discovery file is
removed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
