ericm-db opened a new pull request, #57147: URL: https://github.com/apache/spark/pull/57147
### What changes were proposed in this pull request? Opt-in reuse of a persistent local Spark Connect server for fast local iteration, implemented by driving the standard `sbin/start-connect-server.sh` script. This is the successor of https://github.com/apache/spark/pull/56907, restructured per https://github.com/apache/spark/pull/56907#discussion_r3547739905: instead of a detached Python daemon that boots a classic Py4J session with the Connect plugin and babysits it, PySpark now starts the server through `sbin/start-connect-server.sh` (`spark-daemon.sh submit SparkConnectServer`) and lets `spark-daemon.sh` own the process (daemonization, pid file, logs). The branch stacks on #56907, so the incremental diff between the two approaches is visible here: https://github.com/ericm-db/spark/compare/local-connect-reuse...local-connect-reuse-sbin (net -313 lines). Mechanics, all in `python/pyspark/sql/connect/local_server.py`: - When `spark.local.connect.reuse` / `SPARK_LOCAL_CONNECT_REUSE` is set and the master is `local[...]`, `SparkSession.builder` calls `reuse_or_start_local_connect_server()` instead of booting a fresh in-process server. - The first run launches `sbin/start-connect-server.sh` with the auth token in its environment (never argv), waits for the port, and writes a `0600` *discovery file* (host, port, token, pid from the spark-daemon pid file, Spark version) under `~/.spark/` (relocatable via `SPARK_LOCAL_CONNECT_DISCOVERY`). - Later runs validate the record (matching Spark version, live pid, port accepting connections) and reconnect in a fraction of a second; anything stale is rejected and a fresh server started. First start-up is serialized across processes with a file lock. - Each run connects as its own Connect session, so session-local state does not leak between runs (`SparkConnectServer` enables artifact isolation itself). - The server runs until stopped: `python -m pyspark.sql.connect.local_server --stop` or `sbin/stop-connect-server.sh`. Server logs land next to the discovery file (`~/.spark/logs`). - User start-up confs from the first run are forwarded via a `0600` spark-submit `--properties-file`. - The pip package now ships `start-connect-server.sh` / `stop-connect-server.sh` (their dependencies, `bin/*` and `spark-daemon.sh`, were already packaged). - The reuse path relies on the POSIX `sbin/` scripts and raises a clear error on Windows. ### Why are the changes needed? `SparkSession.builder.remote("local[*]").getOrCreate()` boots a new JVM and Connect server in every Python process, so each `python script.py` iteration re-pays several seconds of cold start. Reconnecting to one long-lived local server makes the edit/run loop sub-second. This is off by default and adds no new server mechanism: it automates exactly the documented manual workflow (`start-connect-server.sh` + `.remote("sc://localhost:15002")`). ### Does this PR introduce _any_ user-facing change? No change unless explicitly opted in. When opted in (new `spark.local.connect.reuse` conf / `SPARK_LOCAL_CONNECT_REUSE` env var, plus optional `spark.local.connect.server.port`), `.remote("local[*]")` reconnects to a persistent local server instead of booting an in-process one. New `python -m pyspark.sql.connect.local_server --stop` command. pip installs additionally ship the connect start/stop sbin scripts. Documented in `spark-connect-overview.md`. ### How was this patch tested? New test suite `pyspark.sql.tests.connect.test_connect_local_server` (19 tests, wired into the `pyspark_connect` module): discovery-file validation (missing/malformed/typed fields), reuse decision (version mismatch, dead pid, closed port, live server; the pid probe is asserted to never run on Windows, where `os.kill(pid, 0)` terminates the target), port picking, seed properties file permissions/format, start-lock round trip, stop with and without a running server, the `--stop` CLI, and POSIX gating. End-to-end tests start real `spark-daemon.sh`-managed servers: builder opt-in, three concurrent cold-starting clients converging on a single server, reconnect + server-side session isolation, and static conf seeding. ``` python/run-tests --testnames pyspark.sql.tests.connect.test_connect_local_server ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-fable-5) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
