This is an automated email from the ASF dual-hosted git repository.

gaogaotiantian pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 73fa0829f7e8 [SPARK-57787][CONNECT] Reuse a persistent local Spark 
Connect server for faster local startup
73fa0829f7e8 is described below

commit 73fa0829f7e87df77efb5ed2fd2f1d67224754a2
Author: Eric Marnadi <[email protected]>
AuthorDate: Thu Jul 30 13:55:12 2026 -0700

    [SPARK-57787][CONNECT] Reuse a persistent local Spark Connect server for 
faster local startup
    
    ### What changes were proposed in this pull request?
    
    Adds an opt-in fast path for local Spark Connect development. Today
    `SparkSession.builder.remote("local[*]").getOrCreate()` starts a fresh 
in-process Connect server
    that lives only as long as that Python process, so every run re-pays the 
cold start (JVM warmup,
    `SparkContext` + server boot).
    
    When `SPARK_LOCAL_CONNECT_REUSE=1` is set (or 
`spark.local.connect.reuse=true` on the builder), a
    `local`-mode remote session instead reconnects to a persistent local 
Connect server. The first run
    starts one through the standard `sbin/start-connect-server.sh` script 
(daemonized by
    `sbin/spark-daemon.sh`, the same daemon a user would start by hand) and 
records host, port, auth
    token, pid and Spark version in a discovery file. Later runs reuse that 
server if its Spark
    version matches, its pid is alive, and its port accepts connections; 
otherwise they start a fresh
    one. If a live but non-reusable server is still running (e.g. after a Spark 
upgrade), start-up
    fails with an error pointing at the `--stop` command below, so old servers 
cannot silently
    accumulate. User code is unchanged: the first run pays the cold start once, 
later runs reconnect
    in a fraction of a second.
    
    The implementation is `pyspark/sql/connect/local_server.py`, with 
`Discovery` handling the
    discovery file (location, load/save, and the cross-process lock -- 
`Discovery` is a context
    manager and all reads and writes happen under the lock), 
`LocalConnectServer` representing one
    recorded server (its URL, reusability probe, stop), and `ServerLauncher` 
running the sbin script
    and waiting for readiness. The first caller's startup confs are forwarded 
to the new server via a
    `--properties-file`, mirroring what the in-process path does with its 
`SparkConf`.
    
    The feature is off by default, Python-only, and POSIX-only (it relies on 
the `sbin/` shell
    scripts). All state -- discovery file, daemon pid file, logs -- lives in a 
per-user `0700`
    directory under the system temp dir rather than the home directory, so 
nothing accumulates
    across reboots; `SPARK_LOCAL_CONNECT_DISCOVERY` overrides the location. The 
auth token is stored
    with `0600` and the server binds localhost, so other users on the machine 
can neither read the
    token nor reach the server. Each run is its own Connect session, so 
session-local state (temp
    views, runtime SQL confs, artifacts) stays per-run; only shared 
`SparkContext` state (catalog,
    global temp views, cached data) carries across runs.
    
    The server runs until stopped with `python -m 
pyspark.sql.connect.local_server --stop`. It is an
    ordinary `spark-daemon.sh` daemon, but it runs with a per-user pid 
directory and ident string so
    it cannot collide with a manually started server -- which also means a plain
    `sbin/stop-connect-server.sh` does not find it. The `--stop` command 
signals the recorded pid and
    clears the discovery file; killing the pid directly also works, and the 
next run notices the dead
    server and starts a fresh one. The earlier idle-timeout self-reaping was 
dropped with the switch
    to the standard sbin daemon; a bounded lifetime would belong server-side 
(e.g. in
    `SparkConnectService`), which also fits the follow-up discussed below of 
moving the
    discovery-file write path into the server so any language client (and the 
proposed `spark
    connect` CLI) can use the same mechanism.
    
    ### Why are the changes needed?
    
    Creating a local Spark session for a quick edit/run loop takes a few 
seconds, and that cost is
    one-time-per-process -- it does not amortize across separate runs. Keeping 
a warm server alive and
    reconnecting to it is the only way to make a repeated local dev/test loop 
fast. This makes that
    behavior available behind a single opt-in, without changing user code or 
default behavior.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Only when the opt-in is enabled. With `SPARK_LOCAL_CONNECT_REUSE=1` (or
    `spark.local.connect.reuse=true` on the builder),
    `SparkSession.builder.remote("local[*]").getOrCreate()` starts a persistent 
local Connect server on
    the first run and reconnects on later runs, instead of booting a fresh 
in-process server each time.
    With the opt-in unset (the default), behavior is unchanged. A new 
documentation section describes
    the manual persistent-server workflow.
    
    ### How was this patch tested?
    
    New `python/pyspark/sql/tests/connect/test_connect_local_server.py`. Unit 
tests cover the
    `Discovery` save/load round-trip, file permissions and malformed-input 
rejection, the
    `is_reusable` decision (version mismatch, dead pid, listening, closed port, 
and the Windows
    pid-probe guard), stop semantics, the `--stop` CLI, and the POSIX guard. 
Four end-to-end tests
    start a real server via the sbin scripts: builder integration with the 
opt-in, three concurrent
    first-time startups converging on one server, reuse across calls with 
session isolation between
    two connections, and the first caller's startup confs reaching the server's 
`SparkConf`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Opus 4.8, Fable 5)
    
    Closes #56907 from ericm-db/local-connect-reuse.
    
    Authored-by: Eric Marnadi <[email protected]>
    Signed-off-by: Tian Gao <[email protected]>
    (cherry picked from commit 0366e48951db36afcf305d49138b29716e729728)
    Signed-off-by: Tian Gao <[email protected]>
---
 dev/sparktestsupport/modules.py                    |   1 +
 docs/spark-connect-overview.md                     |  68 ++-
 python/packaging/classic/setup.py                  |   2 +
 python/pyspark/errors/error-conditions.json        |  10 +
 python/pyspark/sql/connect/local_server.py         | 498 +++++++++++++++++++++
 python/pyspark/sql/session.py                      |  21 +-
 .../sql/tests/connect/test_connect_local_server.py | 381 ++++++++++++++++
 7 files changed, 979 insertions(+), 2 deletions(-)

diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 77aa43eb8d94..898a3831f484 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -1178,6 +1178,7 @@ pyspark_connect = Module(
         "pyspark.sql.tests.connect.test_connect_readwriter",
         "pyspark.sql.tests.connect.test_connect_retry",
         "pyspark.sql.tests.connect.test_connect_session",
+        "pyspark.sql.tests.connect.test_connect_local_server",
         "pyspark.sql.tests.connect.test_connect_stat",
         "pyspark.sql.tests.connect.test_parity_geographytype",
         "pyspark.sql.tests.connect.test_parity_geometrytype",
diff --git a/docs/spark-connect-overview.md b/docs/spark-connect-overview.md
index 4d455a8e4680..2f1c0836c383 100644
--- a/docs/spark-connect-overview.md
+++ b/docs/spark-connect-overview.md
@@ -277,6 +277,72 @@ The connection may also be programmatically created using 
_SparkSession#builder_
 </div>
 </div>
 
+## Faster local iteration with a persistent Connect server
+
+When you develop or test locally with
+
+```python
+from pyspark.sql import SparkSession
+spark = SparkSession.builder.remote("local[*]").getOrCreate()
+```
+
+PySpark boots a fresh in-process Spark Connect server that lives only as long 
as that Python
+process. Every `python script.py` invocation (or every new test worker 
process) therefore re-pays
+the one-time startup cost -- JVM warmup, `SparkContext` construction, and 
Connect server boot --
+which can take a few seconds and makes a quick edit/run loop feel slow.
+
+To amortize that cost across runs, start one persistent local Spark Connect 
server and point
+every run at it:
+
+```bash
+# Start once; it stays up across runs. (--master is optional; it defaults to 
local[*].)
+$SPARK_HOME/sbin/start-connect-server.sh --master "local[*]"
+
+# Every run reconnects instead of booting a new server.
+python -c 'from pyspark.sql import SparkSession; 
SparkSession.builder.remote("sc://localhost:15002").getOrCreate()'
+
+# Stop it when you are done.
+$SPARK_HOME/sbin/stop-connect-server.sh
+```
+
+Alternatively, on POSIX systems PySpark can manage this persistent server for 
you. With
+`SPARK_LOCAL_CONNECT_REUSE=1` set (or `spark.local.connect.reuse=true` on the 
builder),
+`SparkSession.builder.remote("local[*]").getOrCreate()` starts a persistent 
server through
+`sbin/start-connect-server.sh` on the first run and reconnects to it on later 
runs, so scripts
+keep the plain `local[*]` URL:
+
+```bash
+export SPARK_LOCAL_CONNECT_REUSE=1
+
+# The first run starts the server; later runs reconnect to it.
+python -c 'from pyspark.sql import SparkSession; 
SparkSession.builder.remote("local[*]").getOrCreate()'
+
+# Stop the managed server when you are done.
+python -m pyspark.sql.connect.local_server --stop
+```
+
+The managed server is an ordinary `spark-daemon.sh` daemon, but it runs with a 
per-user pid
+directory and ident string so it cannot collide with a server you started by 
hand -- which also
+means a plain `sbin/stop-connect-server.sh` does not find it. The `--stop` 
command signals the
+recorded server and cleans up the discovery file; killing the server's pid 
directly also works,
+and the next run notices the dead server and starts a fresh one.
+
+This managed-server workflow is experimental. The `--stop` command and the 
discovery file
+location and format may change in a future release, for example if local 
server management is
+folded into a unified `spark connect` CLI.
+
+The connection details (host, port, auth token, pid, Spark version) are 
recorded in a discovery
+file in a private per-user directory; set `SPARK_LOCAL_CONNECT_DISCOVERY` to 
override its
+location. A run reconnects only to a server whose Spark version matches. After 
upgrading Spark,
+the server from the previous version cannot be reused and the next run fails 
with an error asking
+you to stop it; run the `--stop` command above and rerun to start a fresh 
server.
+
+Each run connects as its own Connect session, so session-local state -- temp 
views, runtime SQL
+configurations, and session artifacts -- is fresh on every run and never leaks 
between runs. State
+backed by the shared `SparkContext` (the persistent catalog/warehouse, global 
temp views, and
+cached datasets) *is* shared across runs, so namespace per-run databases or 
clear that state
+yourself if your runs must be fully isolated.
+
 ## Use Spark Connect in standalone applications
 
 <div class="codetabs">
@@ -371,7 +437,7 @@ one may implement their own class extending `ClassFinder` 
for customized search
 </div>
 
 For more information on application development with Spark Connect as well as 
extending Spark Connect
-with custom functionality, see [Application Development with Spark 
Connect](app-dev-spark-connect.html). 
+with custom functionality, see [Application Development with Spark 
Connect](app-dev-spark-connect.html).
 # Client application authentication
 
 While Spark Connect does not have built-in authentication, it is designed to
diff --git a/python/packaging/classic/setup.py 
b/python/packaging/classic/setup.py
index 97ec9dd90ddb..5ec33f405d49 100755
--- a/python/packaging/classic/setup.py
+++ b/python/packaging/classic/setup.py
@@ -345,7 +345,9 @@ try:
             "pyspark.sbin": [
                 "spark-config.sh",
                 "spark-daemon.sh",
+                "start-connect-server.sh",
                 "start-history-server.sh",
+                "stop-connect-server.sh",
                 "stop-history-server.sh",
             ],
             "pyspark.python.lib": ["*.zip"],
diff --git a/python/pyspark/errors/error-conditions.json 
b/python/pyspark/errors/error-conditions.json
index b8bbaf097279..bd1e5fe9ad2a 100644
--- a/python/pyspark/errors/error-conditions.json
+++ b/python/pyspark/errors/error-conditions.json
@@ -546,6 +546,16 @@
       "<arg1> and <arg2> should be of the same length, got <arg1_length> and 
<arg2_length>."
     ]
   },
+  "LOCAL_CONNECT_RUNTIME_DIR_UNAVAILABLE": {
+    "message": [
+      "Cannot claim the per-user runtime directory <path> (was it created by 
another user?); remove it or point SPARK_LOCAL_CONNECT_DISCOVERY at a path you 
own."
+    ]
+  },
+  "LOCAL_CONNECT_SERVER_START_FAILED": {
+    "message": [
+      "Failed to start a persistent local Spark Connect server: <reason>."
+    ]
+  },
   "LOCAL_RELATION_SIZE_LIMIT_EXCEEDED": {
     "message": [
       "Local relation size (<actualSize> bytes) exceeds the limit (<sizeLimit> 
bytes)."
diff --git a/python/pyspark/sql/connect/local_server.py 
b/python/pyspark/sql/connect/local_server.py
new file mode 100644
index 000000000000..4f06d884aff4
--- /dev/null
+++ b/python/pyspark/sql/connect/local_server.py
@@ -0,0 +1,498 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+Opt-in reuse of a persistent local Spark Connect server 
(``spark.local.connect.reuse`` /
+``SPARK_LOCAL_CONNECT_REUSE``).
+
+By default ``SparkSession.builder.remote("local[*]").getOrCreate()`` boots a 
fresh in-process
+Connect server in every Python process. With reuse enabled, the first run 
starts one
+long-lived server through ``sbin/start-connect-server.sh`` and records how to 
reach it (host,
+port, auth token, pid, Spark version) in a discovery file; later runs 
reconnect to it if the
+version matches, the pid is alive, and the port accepts connections. Each run 
still gets its
+own server-side session, so session-local state does not leak between runs.
+
+The discovery file, the daemon's pid file, and the logs live in a per-user 
``0700`` directory
+under the system temp dir; ``SPARK_LOCAL_CONNECT_DISCOVERY`` overrides the 
discovery file
+location. The auth token is stored with ``0600`` and the server binds 
localhost, so other
+users on the machine can neither read the token nor reach the server. 
Processes of the same
+user share the server by design.
+
+The server runs until stopped with ``python -m 
pyspark.sql.connect.local_server --stop``.
+(A plain ``sbin/stop-connect-server.sh`` cannot find it: the daemon runs with 
a custom pid
+dir and ident string.) Windows is not supported, as this relies on the POSIX 
scripts under
+``sbin/``.
+
+This module is experimental. The discovery file location and format and the 
``--stop``
+entry point are internal details that may change or move server-side (e.g. 
into a unified
+``spark connect`` CLI); only the reuse opt-in itself is meant to be a stable 
surface.
+"""
+
+import argparse
+import contextlib
+import getpass
+import json
+import os
+import signal
+import socket
+import subprocess
+import sys
+import tempfile
+import time
+import uuid
+from typing import Any, Dict, Iterator, Optional, TextIO
+
+from pyspark.errors import PySparkRuntimeError
+
+_SERVER_CLASS = "org.apache.spark.sql.connect.service.SparkConnectServer"
+# A fixed SPARK_IDENT_STRING keeps the spark-daemon.sh pid and log file names 
stable
+# regardless of $USER.
+_SPARK_IDENT = "local-connect"
+
+
+def _pid_alive(pid: int) -> bool:
+    """Whether ``pid`` exists (POSIX only). A process we cannot signal counts 
as alive."""
+    try:
+        os.kill(pid, 0)
+    except ProcessLookupError:
+        return False
+    except OSError:
+        pass
+    return True
+
+
+class Discovery:
+    """Reads and writes the discovery file recording the persistent local 
server.
+
+    The file lives in a per-user directory under the system temp dir, or 
wherever
+    ``SPARK_LOCAL_CONNECT_DISCOVERY`` points; the daemon's pid file and logs 
sit next to it.
+    """
+
+    @staticmethod
+    def _runtime_dir() -> str:
+        path = os.path.join(tempfile.gettempdir(), 
"spark-connect-{}".format(getpass.getuser()))
+        try:
+            # exist_ok also covers two first runs racing to create the 
directory; chmod
+            # re-asserts 0700 and fails if another user owns the path.
+            os.makedirs(path, mode=0o700, exist_ok=True)
+            os.chmod(path, 0o700)
+        except OSError as e:
+            raise PySparkRuntimeError(
+                errorClass="LOCAL_CONNECT_RUNTIME_DIR_UNAVAILABLE",
+                messageParameters={"path": path},
+            ) from e
+        return path
+
+    def __init__(self, path: Optional[str] = None):
+        self.path = os.path.abspath(
+            path
+            or os.environ.get("SPARK_LOCAL_CONNECT_DISCOVERY")
+            or os.path.join(self._runtime_dir(), "connect-local.json")
+        )
+        self._lock_file: Optional[TextIO] = None
+
+    @property
+    def directory(self) -> str:
+        return os.path.dirname(self.path)
+
+    @property
+    def daemon_pid_path(self) -> str:
+        return os.path.join(self.directory, 
"spark-{}-{}-1.pid".format(_SPARK_IDENT, _SERVER_CLASS))
+
+    def daemon_pid(self) -> Optional[int]:
+        """The pid recorded by ``spark-daemon.sh``, or ``None`` if absent or 
unreadable.
+        The daemon writes this file outside our lock, so no lock is required 
to read it.
+        """
+        try:
+            with open(self.daemon_pid_path, "r") as f:
+                return int(f.read().strip())
+        except (OSError, ValueError):
+            return None
+
+    def __enter__(self) -> "Discovery":
+        os.makedirs(self.directory, exist_ok=True)
+        self._lock_file = open(self.path + ".lock", "a+")
+        import fcntl
+
+        fcntl.flock(self._lock_file.fileno(), fcntl.LOCK_EX)
+        return self
+
+    def __exit__(self, exc_type: Any, exc_value: Any, traceback: Any) -> None:
+        assert self._lock_file is not None
+        self._lock_file.close()
+        self._lock_file = None
+
+    def _assert_locked(self) -> None:
+        assert self._lock_file is not None, "Discovery must be used as a 
context manager"
+
+    def load(self) -> Optional[Dict[str, Any]]:
+        """Read the discovery file, returning ``None`` if it is absent or 
malformed."""
+        self._assert_locked()
+        try:
+            with open(self.path, "r") as f:
+                data = json.load(f)
+        except (OSError, ValueError):
+            return None
+        if not isinstance(data, dict):
+            return None
+        try:
+            data["port"] = int(data["port"])
+            data["pid"] = int(data["pid"])
+        except (KeyError, TypeError, ValueError):
+            return None
+        if not all(isinstance(data[k], str) for k in ("host", "token", 
"spark_version")):
+            return None
+        return data
+
+    def save(self, data: Dict[str, Any]) -> None:
+        """Write the discovery file with ``0600`` perms; it holds the auth 
token. Readers
+        and writers all hold the exclusive lock, so the write does not need to 
be atomic.
+        """
+        self._assert_locked()
+        fd = os.open(self.path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+        with os.fdopen(fd, "w") as f:
+            # O_CREAT applies the 0600 mode only when it creates the file; 
re-assert it in
+            # case a pre-existing file had wider permissions.
+            os.fchmod(fd, 0o600)
+            f.write(json.dumps(data))
+
+    def clear(self) -> None:
+        self._assert_locked()
+        with contextlib.suppress(OSError):
+            os.remove(self.path)
+        with contextlib.suppress(OSError):
+            os.remove(self.daemon_pid_path)
+
+
+class LocalConnectServer:
+    """The persistent server described by a locked ``Discovery``."""
+
+    def __init__(self, discovery: Discovery):
+        self._discovery = discovery
+        self._reload()
+
+    def _reload(self) -> None:
+        # ``data`` is None when no server is recorded yet (first run, or the 
file was
+        # cleared): the None fields make ``is_reusable()`` False, so 
``reuse_or_start()``
+        # launches a fresh server.
+        data = self._discovery.load()
+        self.host = data["host"] if data else None
+        self.port = data["port"] if data else None
+        self.token = data["token"] if data else None
+        self.pid = data["pid"] if data else None
+        self.spark_version = data["spark_version"] if data else None
+
+    @property
+    def url(self) -> str:
+        assert self.host is not None and self.port is not None
+        return "sc://{}:{}".format(self.host, self.port)
+
+    def is_listening(self) -> bool:
+        if self.host is None or self.port is None:
+            return False
+        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
+            sock.settimeout(0.5)
+            return sock.connect_ex((self.host, self.port)) == 0
+
+    def is_reusable(self) -> bool:
+        from pyspark.version import __version__
+
+        if self.spark_version != __version__ or self.pid is None:
+            return False
+        if os.name == "posix" and not _pid_alive(self.pid):
+            return False
+        return self.is_listening()
+
+    def reuse_or_start(self, master: str, opts: Dict[str, Any]) -> str:
+        if not self.is_reusable():
+            ServerLauncher(master, opts, self._discovery).launch()
+            self._reload()
+        assert self.token is not None
+        os.environ["SPARK_CONNECT_AUTHENTICATE_TOKEN"] = self.token
+        return self.url
+
+    def stop(self) -> bool:
+        stopped = False
+        if self.pid is not None:
+            try:
+                os.kill(self.pid, signal.SIGTERM)
+                stopped = True
+            except OSError:
+                pass
+        self._discovery.clear()
+        return stopped
+
+
+class ServerLauncher:
+    """Starts a persistent local server via ``sbin/start-connect-server.sh`` 
and waits until
+    it accepts connections. Callers must hold a ``Discovery`` context.
+    """
+
+    _READY_TIMEOUT = 120
+
+    def __init__(self, master: str, opts: Dict[str, Any], discovery: 
Discovery):
+        self._master = master
+        self._opts = opts
+        self._discovery = discovery
+        self._log_dir = os.path.join(discovery.directory, "logs")
+
+    def launch(self) -> None:
+        token = self._token()
+        port = self._pick_port()
+        # The conf file must outlive _await_ready: spark-daemon.sh backgrounds 
the JVM,
+        # which reads --properties-file while starting up.
+        with self._seed_properties_file() as conf_file:
+            self._run_script(port, token, conf_file)
+            self._await_ready(port, token)
+
+    def _token(self) -> str:
+        # Same precedence as the in-process _start_connect_server: explicit 
env token, then
+        # conf, then a fresh one. Passed via the environment so it never shows 
up in `ps`.
+        return (
+            os.environ.get("SPARK_CONNECT_AUTHENTICATE_TOKEN")
+            or self._opts.get("spark.connect.authenticate.token")
+            or str(uuid.uuid4())
+        )
+
+    def _pick_port(self) -> int:
+        """Under SPARK_TESTING always use an OS-assigned free port so suites 
can run in
+        parallel; otherwise honor the configured/default port, falling back to 
a free one if
+        another process holds it. (A live stale server of ours also holds the 
port, but that
+        start fails later at spark-daemon.sh's pid-file check regardless of 
port.) The sbin
+        script cannot report an ephemeral port back, so the free port is 
picked and released
+        here, with a small race until the server binds it.
+        """
+
+        def free_port() -> int:
+            with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
+                sock.bind(("localhost", 0))
+                return sock.getsockname()[1]
+
+        if "SPARK_TESTING" in os.environ:
+            return free_port()
+        from pyspark.sql.connect.client import DefaultChannelBuilder
+
+        port = int(
+            self._opts.get("spark.local.connect.server.port", 
DefaultChannelBuilder.default_port())
+        )
+        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
+            try:
+                sock.bind(("localhost", port))
+                return port
+            except OSError:
+                return free_port()
+
+    def _seed_conf(self) -> Dict[str, Any]:
+        """Startup confs for the new server, merged like the in-process
+        ``_start_connect_server`` merges ``PYSPARK_REMOTE_INIT_CONF_*`` with 
the builder
+        opts, minus the keys the launcher sets itself and the 
``spark.local.connect.*``
+        opt-in keys. Only the run that starts the server can seed static 
confs; later runs
+        find the JVM already warm.
+        """
+        conf: Dict[str, Any] = {}
+        for i in range(int(os.environ.get("PYSPARK_REMOTE_INIT_CONF_LEN", 
"0"))):
+            conf = 
json.loads(os.environ["PYSPARK_REMOTE_INIT_CONF_{}".format(i)])
+        conf.update(self._opts)
+        for k in list(conf):
+            if k in (
+                "spark.remote",
+                "spark.api.mode",
+                "spark.master",
+                "spark.connect.authenticate.token",
+                "spark.connect.grpc.binding.port",
+            ) or k.startswith("spark.local.connect."):
+                conf.pop(k)
+        return conf
+
+    @contextlib.contextmanager
+    def _seed_properties_file(self) -> Iterator[Optional[str]]:
+        # NamedTemporaryFile creates the file with 0600 perms since confs may 
hold sensitive
+        # values; a --properties-file keeps them off the server's argv where 
they would show
+        # up in `ps`. Yields None when there is nothing to seed.
+        seed = self._seed_conf()
+        if not seed:
+            yield None
+            return
+        with tempfile.NamedTemporaryFile(
+            mode="w",
+            prefix="connect-local-conf-",
+            suffix=".properties",
+            dir=self._discovery.directory,
+        ) as f:
+            for key, value in seed.items():
+                escaped = str(value).replace("\\", "\\\\").replace("\n", "\\n")
+                f.write("{}={}\n".format(key, escaped))
+            f.flush()
+            yield f.name
+
+    def _run_script(self, port: int, token: str, conf_file: Optional[str]) -> 
None:
+        from pyspark.find_spark_home import _find_spark_home
+
+        spark_home = os.environ.get("SPARK_HOME") or _find_spark_home()
+        script = os.path.join(spark_home, "sbin", "start-connect-server.sh")
+        if not os.path.isfile(script):
+            raise PySparkRuntimeError(
+                errorClass="LOCAL_CONNECT_SERVER_START_FAILED",
+                messageParameters={"reason": "cannot find {}".format(script)},
+            )
+
+        env = dict(os.environ)
+        for var in ("SPARK_REMOTE", "SPARK_LOCAL_REMOTE", 
"SPARK_CONNECT_MODE_ENABLED"):
+            env.pop(var, None)
+        env["SPARK_CONNECT_AUTHENTICATE_TOKEN"] = token
+        env["SPARK_PID_DIR"] = self._discovery.directory
+        env["SPARK_LOG_DIR"] = self._log_dir
+        env["SPARK_IDENT_STRING"] = _SPARK_IDENT
+
+        cmd = [
+            script,
+            "--master",
+            self._master,
+            "--conf",
+            "spark.connect.grpc.binding.port={}".format(port),
+        ]
+        if conf_file is not None:
+            cmd += ["--properties-file", conf_file]
+
+        result = subprocess.run(
+            cmd,
+            env=env,
+            stdin=subprocess.DEVNULL,
+            capture_output=True,
+            text=True,
+            timeout=120,
+        )
+        if result.returncode != 0:
+            stale_pid = self._discovery.daemon_pid()
+            if stale_pid is not None and _pid_alive(stale_pid):
+                # spark-daemon.sh refuses to start while its pid file points 
at a live
+                # process -- here a server this client just rejected as not 
reusable
+                # (e.g. after a Spark upgrade).
+                raise PySparkRuntimeError(
+                    errorClass="LOCAL_CONNECT_SERVER_START_FAILED",
+                    messageParameters={
+                        "reason": "a local Connect server that is not reusable 
by this client is "
+                        "already running (pid {}); stop it with "
+                        "`python -m pyspark.sql.connect.local_server 
--stop`".format(stale_pid)
+                    },
+                )
+            output = (result.stderr or "") + (result.stdout or "")
+            last_line = output.strip().splitlines()[-1] if output.strip() else 
""
+            raise PySparkRuntimeError(
+                errorClass="LOCAL_CONNECT_SERVER_START_FAILED",
+                messageParameters={
+                    "reason": "start-connect-server.sh exited with code {}: 
{}".format(
+                        result.returncode, last_line
+                    )
+                },
+            )
+
+    def _await_ready(self, port: int, token: str) -> None:
+        from pyspark.version import __version__
+
+        deadline = time.time() + self._READY_TIMEOUT
+        while time.time() < deadline:
+            pid = self._discovery.daemon_pid()
+            if pid is not None:
+                with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
+                    sock.settimeout(0.5)
+                    listening = sock.connect_ex(("localhost", port)) == 0
+                if listening:
+                    self._discovery.save(
+                        {
+                            "host": "localhost",
+                            "port": port,
+                            "token": token,
+                            "pid": pid,
+                            "spark_version": __version__,
+                        }
+                    )
+                    return
+                if not _pid_alive(pid):
+                    raise PySparkRuntimeError(
+                        errorClass="LOCAL_CONNECT_SERVER_START_FAILED",
+                        messageParameters={
+                            "reason": "the server exited during start-up; see 
logs under {}".format(
+                                self._log_dir
+                            )
+                        },
+                    )
+            time.sleep(0.25)
+
+        pid = self._discovery.daemon_pid()
+        if pid is not None:
+            with contextlib.suppress(OSError):
+                os.kill(pid, signal.SIGTERM)
+        raise PySparkRuntimeError(
+            errorClass="LOCAL_CONNECT_SERVER_START_FAILED",
+            messageParameters={
+                "reason": "the server did not become ready within {} seconds; "
+                "see logs under {}".format(self._READY_TIMEOUT, self._log_dir)
+            },
+        )
+
+
+def reuse_or_start_local_connect_server(master: str, opts: Dict[str, Any]) -> 
str:
+    """Reuse a running persistent local Connect server, or start one if none 
is reusable.
+
+    Returns the ``sc://host:port`` endpoint and sets 
``SPARK_CONNECT_AUTHENTICATE_TOKEN`` so
+    the client authenticates against that server. Only reached for a ``local`` 
master when
+    the reuse opt-in is set; see ``SparkSession.getOrCreate`` in 
``pyspark.sql.session``.
+    """
+    if os.name != "posix":
+        raise PySparkRuntimeError(
+            errorClass="LOCAL_CONNECT_SERVER_START_FAILED",
+            messageParameters={
+                "reason": "spark.local.connect.reuse relies on the POSIX 
scripts under sbin/; "
+                "on this platform start a server manually 
(sbin/start-connect-server.sh) and "
+                'connect with .remote("sc://...")'
+            },
+        )
+    with Discovery() as discovery:
+        return LocalConnectServer(discovery).reuse_or_start(master, opts)
+
+
+def stop_local_connect_server() -> bool:
+    """Stop the recorded persistent local Connect server, if any; safe to call 
when none is
+    running. Also available as ``python -m pyspark.sql.connect.local_server 
--stop``.
+    """
+    with Discovery() as discovery:
+        return LocalConnectServer(discovery).stop()
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Manage the persistent local Spark Connect server used by 
the opt-in "
+        "spark.local.connect.reuse path. The server itself is started on 
demand through "
+        "sbin/start-connect-server.sh."
+    )
+    parser.add_argument(
+        "--stop", action="store_true", help="stop the recorded running server, 
if any"
+    )
+    args = parser.parse_args()
+
+    if not args.stop:
+        parser.print_help(sys.stderr)
+        sys.exit(2)
+    if stop_local_connect_server():
+        print("Stopped the persistent local Spark Connect server.")
+    else:
+        print("No running persistent local Spark Connect server found.")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/python/pyspark/sql/session.py b/python/pyspark/sql/session.py
index c36260b9d13e..e9398bd51278 100644
--- a/python/pyspark/sql/session.py
+++ b/python/pyspark/sql/session.py
@@ -523,7 +523,26 @@ class SparkSession(SparkConversionMixin):
                                     messageParameters={},
                                 )
 
-                            if url.startswith("local") or (
+                            reuse_local = str(
+                                opts.get(
+                                    "spark.local.connect.reuse",
+                                    
os.environ.get("SPARK_LOCAL_CONNECT_REUSE", ""),
+                                )
+                            ).lower() in ("1", "true")
+
+                            if url.startswith("local") and reuse_local:
+                                from pyspark.sql.connect.local_server import (
+                                    reuse_or_start_local_connect_server,
+                                )
+
+                                # Opt-in: reconnect to a persistent local 
Connect server (starting
+                                # one on the first run) instead of booting a 
fresh in-process server
+                                # every process. See 
`pyspark.sql.connect.local_server`.
+                                url = reuse_or_start_local_connect_server(url, 
opts)
+                                for k in list(opts):
+                                    if k.startswith("spark.local.connect."):
+                                        opts.pop(k)
+                            elif url.startswith("local") or (
                                 is_api_mode_connect and not 
url.startswith("sc://")
                             ):
                                 os.environ["SPARK_LOCAL_REMOTE"] = "1"
diff --git a/python/pyspark/sql/tests/connect/test_connect_local_server.py 
b/python/pyspark/sql/tests/connect/test_connect_local_server.py
new file mode 100644
index 000000000000..eba26972ae8c
--- /dev/null
+++ b/python/pyspark/sql/tests/connect/test_connect_local_server.py
@@ -0,0 +1,381 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import contextlib
+import getpass
+import json
+import os
+import shutil
+import signal
+import socket
+import subprocess
+import sys
+import tempfile
+import textwrap
+import time
+import unittest
+
+from pyspark.util import is_remote_only
+from pyspark.testing.connectutils import should_test_connect, 
connect_requirement_message
+
+if should_test_connect:
+    from pyspark.sql import SparkSession as PySparkSession
+    from pyspark.sql.connect import local_server
+    from pyspark.sql.connect.local_server import Discovery, LocalConnectServer
+    from pyspark.sql.connect.session import SparkSession as RemoteSparkSession
+    from pyspark.version import __version__
+
+
[email protected]
+def _listening_socket():
+    listener = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    try:
+        listener.bind(("localhost", 0))
+        listener.listen(1)
+        yield listener.getsockname()[1]
+    finally:
+        listener.close()
+
+
[email protected](
+    not should_test_connect or is_remote_only(),
+    connect_requirement_message or "Requires JVM access to start a local 
Connect server",
+)
+class LocalConnectServerReuseTests(unittest.TestCase):
+    """Tests for the opt-in persistent local Spark Connect server 
(SPARK_LOCAL_CONNECT_REUSE)."""
+
+    def setUp(self) -> None:
+        # Point discovery at a throwaway path so the real per-user file is 
never touched.
+        self._tmpdir = tempfile.mkdtemp()
+        self._discovery_path = os.path.join(self._tmpdir, "connect-local.json")
+        self._saved_env = {
+            k: os.environ.get(k)
+            for k in ("SPARK_LOCAL_CONNECT_DISCOVERY", 
"SPARK_CONNECT_AUTHENTICATE_TOKEN")
+        }
+        os.environ["SPARK_LOCAL_CONNECT_DISCOVERY"] = self._discovery_path
+
+    def tearDown(self) -> None:
+        try:
+            # Only stop a real, separately-spawned server. Several tests 
fabricate discovery
+            # files pointing at this very process, which must never be 
signalled.
+            server = self._discovered_server()
+            if server.pid is not None and server.pid != os.getpid():
+                local_server.stop_local_connect_server()
+                # Wait for the JVM to release the port so the next test starts 
clean.
+                self._wait_port_closed(server.host, server.port)
+        finally:
+            for k, v in self._saved_env.items():
+                if v is None:
+                    os.environ.pop(k, None)
+                else:
+                    os.environ[k] = v
+            shutil.rmtree(self._tmpdir, ignore_errors=True)
+
+    def _server(self, **overrides) -> "LocalConnectServer":
+        from unittest import mock
+
+        fields = {
+            "host": "localhost",
+            "port": 0,
+            "token": "t",
+            "pid": os.getpid(),
+            "spark_version": __version__,
+        }
+        fields.update(overrides)
+        discovery = mock.Mock()
+        discovery.load.return_value = fields
+        return LocalConnectServer(discovery)
+
+    def _discovered_server(self) -> "LocalConnectServer":
+        with Discovery() as discovery:
+            return LocalConnectServer(discovery)
+
+    def test_discovery_location(self) -> None:
+        self.assertEqual(Discovery().path, self._discovery_path)
+        # Without the override the file lives in a per-user 0700 dir under the 
temp dir.
+        os.environ.pop("SPARK_LOCAL_CONNECT_DISCOVERY")
+        default = Discovery()
+        self.assertTrue(default.directory.startswith(tempfile.gettempdir()))
+        if os.name == "posix":
+            self.assertIn("spark-connect-{}".format(getpass.getuser()), 
default.directory)
+            self.assertEqual(os.stat(default.directory).st_mode & 0o777, 0o700)
+
+    def test_discovery_roundtrip(self) -> None:
+        with Discovery() as discovery:
+            saved = self._server(port=15002)
+            discovery.save(
+                {k: getattr(saved, k) for k in ("host", "port", "token", 
"pid", "spark_version")}
+            )
+            # The file holds the auth token and must not be readable by other 
users.
+            self.assertEqual(os.stat(discovery.path).st_mode & 0o777, 0o600)
+            loaded = LocalConnectServer(discovery)
+            for attr in ("host", "port", "token", "pid", "spark_version", 
"url"):
+                self.assertEqual(getattr(loaded, attr), getattr(saved, attr), 
attr)
+            discovery.clear()
+            self.assertIsNone(discovery.load())
+            discovery.clear()  # clearing again is a no-op
+
+    def test_discovery_load_rejects_malformed_files(self) -> None:
+        malformed = [
+            "not json",
+            json.dumps(["a", "list"]),
+            json.dumps({"host": "localhost"}),  # missing required keys
+            json.dumps(
+                {
+                    "host": "localhost",
+                    "port": 1,
+                    "token": "t",
+                    "pid": "not-a-pid",
+                    "spark_version": __version__,
+                }
+            ),
+            json.dumps(
+                {"host": None, "port": 1, "token": "t", "pid": 1, 
"spark_version": __version__}
+            ),
+        ]
+        with Discovery() as discovery:
+            for content in malformed:
+                with self.subTest(content=content):
+                    with open(discovery.path, "w") as f:
+                        f.write(content)
+                    self.assertIsNone(discovery.load())
+
+    def test_server_is_reusable(self) -> None:
+        with _listening_socket() as port:
+            with self.subTest("alive process listening on the port with a 
matching version"):
+                self.assertTrue(self._server(port=port).is_reusable())
+            with self.subTest("version mismatch"):
+                self.assertFalse(
+                    self._server(port=port, 
spark_version="0.0.0-not-this-build").is_reusable()
+                )
+            if os.name == "posix":  # the pid probe only runs on POSIX (see 
the test below)
+                with self.subTest("dead pid"):
+                    # PID 2**31 - 1 is effectively guaranteed not to exist.
+                    self.assertFalse(self._server(port=port, pid=2**31 - 
1).is_reusable())
+            server = self._server(port=port)
+        with self.subTest("port no longer listening"):
+            self.assertFalse(server.is_reusable())
+
+    def test_pid_probe_is_skipped_on_windows(self) -> None:
+        # On Windows os.kill(pid, 0) terminates the target instead of probing 
it, so the
+        # reuse check would kill the very server it is examining.
+        from unittest import mock
+
+        with _listening_socket() as port:
+            server = self._server(port=port)
+            with mock.patch.object(os, "name", "nt"), mock.patch.object(os, 
"kill") as kill:
+                self.assertTrue(server.is_reusable())
+                kill.assert_not_called()
+
+    def test_stop_when_no_server_is_safe(self) -> None:
+        self.assertFalse(local_server.stop_local_connect_server())
+
+    def test_stop_signals_recorded_server_and_clears_discovery(self) -> None:
+        from unittest import mock
+
+        with Discovery() as discovery:
+            server = self._server(pid=12345)
+            discovery.save(
+                {k: getattr(server, k) for k in ("host", "port", "token", 
"pid", "spark_version")}
+            )
+        with mock.patch.object(os, "kill") as kill:
+            self.assertTrue(local_server.stop_local_connect_server())
+        kill.assert_called_once_with(12345, signal.SIGTERM)
+        self.assertIsNone(self._discovered_server().pid)
+
+    def test_stop_cli_reports_when_no_server(self) -> None:
+        result = subprocess.run(
+            [sys.executable, "-m", "pyspark.sql.connect.local_server", 
"--stop"],
+            env=dict(os.environ),
+            capture_output=True,
+            text=True,
+            timeout=120,
+        )
+        self.assertEqual(result.returncode, 0, result.stderr)
+        self.assertIn("No running persistent local Spark Connect server", 
result.stdout)
+
+    def test_reuse_or_start_requires_posix(self) -> None:
+        from unittest import mock
+
+        from pyspark.errors import PySparkRuntimeError
+
+        with mock.patch.object(os, "name", "nt"):
+            with self.assertRaises(PySparkRuntimeError) as ctx:
+                local_server.reuse_or_start_local_connect_server("local[2]", 
{})
+        self.assertIn("POSIX", str(ctx.exception))
+
+    def _release(self, session) -> None:
+        """Close one client session without stopping the shared server."""
+        try:
+            session.client.release_session()
+        except Exception:
+            pass
+        try:
+            session.client.close()
+        except Exception:
+            pass
+
+    def _wait_port_closed(self, host, port, timeout=30) -> bool:
+        deadline = time.time() + timeout
+        while time.time() < deadline:
+            with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
+                sock.settimeout(0.5)
+                if sock.connect_ex((host, int(port))) != 0:
+                    return True
+            time.sleep(0.5)
+        return False
+
+    @unittest.skipUnless(os.name == "posix", "the reuse path relies on the 
POSIX sbin scripts")
+    def test_builder_remote_local_uses_reuse_flag(self) -> None:
+        spark = None
+        try:
+            spark = (
+                PySparkSession.builder.remote("local[2]")
+                .config("spark.local.connect.reuse", "true")
+                .getOrCreate()
+            )
+            self.assertEqual(spark.range(2).count(), 2)
+
+            server = self._discovered_server()
+            self.assertIsNotNone(server.pid)
+            self.assertEqual(server.spark_version, __version__)
+            self.assertNotEqual(server.pid, os.getpid())
+        finally:
+            if spark is not None:
+                spark.stop()
+
+    @unittest.skipUnless(os.name == "posix", "the reuse path relies on the 
POSIX sbin scripts")
+    def test_concurrent_startup_reuses_one_server(self) -> None:
+        script = textwrap.dedent("""
+            import json
+            import os
+
+            from pyspark.sql import SparkSession
+
+            spark = (
+                SparkSession.builder.remote("local[2]")
+                .config("spark.local.connect.reuse", "true")
+                .getOrCreate()
+            )
+            try:
+                count = spark.range(1).count()
+                with open(os.environ["SPARK_LOCAL_CONNECT_DISCOVERY"], "r") as 
f:
+                    disc = json.load(f)
+                print(json.dumps({"count": count, "pid": disc["pid"], "port": 
disc["port"]}))
+            finally:
+                spark.stop()
+            """)
+        env = dict(os.environ)
+        env["SPARK_LOCAL_CONNECT_DISCOVERY"] = self._discovery_path
+        env["SPARK_LOCAL_CONNECT_REUSE"] = "1"
+
+        procs = [
+            subprocess.Popen(
+                [sys.executable, "-c", script],
+                env=env,
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE,
+                text=True,
+            )
+            for _ in range(3)
+        ]
+        outputs = []
+        try:
+            for proc in procs:
+                stdout, stderr = proc.communicate(timeout=180)
+                self.assertEqual(proc.returncode, 0, stderr)
+                lines = stdout.strip().splitlines()
+                self.assertTrue(lines, stderr)
+                outputs.append(json.loads(lines[-1]))
+        finally:
+            for proc in procs:
+                if proc.poll() is None:
+                    proc.kill()
+                    proc.communicate()
+
+        self.assertEqual({o["count"] for o in outputs}, {1})
+        self.assertEqual(len({o["pid"] for o in outputs}), 1)
+        self.assertEqual(len({o["port"] for o in outputs}), 1)
+
+    @unittest.skipUnless(os.name == "posix", "the reuse path relies on the 
POSIX sbin scripts")
+    def test_start_reuse_and_session_isolation(self) -> None:
+        endpoint = 
local_server.reuse_or_start_local_connect_server("local[2]", {})
+        self.assertTrue(endpoint.startswith("sc://localhost:"))
+
+        server = self._discovered_server()
+        self.assertIsNotNone(server.pid)
+        self.assertEqual(server.url, endpoint)
+        self.assertEqual(server.spark_version, __version__)
+        self.assertEqual(os.environ.get("SPARK_CONNECT_AUTHENTICATE_TOKEN"), 
server.token)
+        first_pid = server.pid
+
+        s1 = s2 = None
+        try:
+            # A second call reuses the running server instead of spawning a 
new one.
+            endpoint2 = 
local_server.reuse_or_start_local_connect_server("local[2]", {})
+            self.assertEqual(endpoint2, endpoint)
+            self.assertEqual(self._discovered_server().pid, first_pid)
+
+            s1 = RemoteSparkSession.builder.remote(endpoint).create()
+            s2 = RemoteSparkSession.builder.remote(endpoint).create()
+            self.assertEqual(s1.range(5).count(), 5)
+            self.assertEqual(s2.range(3).count(), 3)
+
+            # Session-local state must not leak across connections.
+            s1.range(1).createOrReplaceTempView("only_in_s1")
+            self.assertIn("only_in_s1", [t.name for t in 
s1.catalog.listTables()])
+            self.assertNotIn("only_in_s1", [t.name for t in 
s2.catalog.listTables()])
+        finally:
+            if s1 is not None:
+                self._release(s1)
+            if s2 is not None:
+                self._release(s2)
+
+        self.assertTrue(local_server.stop_local_connect_server())
+        self.assertIsNone(self._discovered_server().pid)
+        # Check the port rather than the pid, which can linger while the JVM 
shuts down.
+        self.assertTrue(
+            self._wait_port_closed(server.host, server.port),
+            "server port {} still open after stop".format(server.port),
+        )
+
+    @unittest.skipUnless(os.name == "posix", "the reuse path relies on the 
POSIX sbin scripts")
+    def test_start_seeds_static_conf_on_the_server(self) -> None:
+        # spark.local.connect.* and spark.master must be stripped from the 
seed, not
+        # forwarded; startup succeeding with them present covers that.
+        warehouse = os.path.join(self._tmpdir, "seeded-wh")
+        opts = {
+            "spark.sql.warehouse.dir": warehouse,
+            "spark.local.connect.reuse": "true",
+            "spark.master": "local[2]",
+        }
+        endpoint = 
local_server.reuse_or_start_local_connect_server("local[2]", opts)
+        spark = None
+        try:
+            spark = RemoteSparkSession.builder.remote(endpoint).create()
+            # A static conf cannot be set per-session after the JVM is up, so 
seeing it here
+            # proves the seed reached the server's SparkConf.
+            
self.assertTrue(spark.conf.get("spark.sql.warehouse.dir").endswith(warehouse))
+        finally:
+            if spark is not None:
+                self._release(spark)
+
+
+if __name__ == "__main__":
+    from pyspark.testing import main
+
+    main()


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to