This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new e847573 [SPARK-61008][CONNECT][PYTHON] Get or create the sessions of
SparkSession.builder as PySpark does
e847573 is described below
commit e8475739b7920312e9d51044e7f5486095d7352d
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 22:29:52 2026 +0900
[SPARK-61008][CONNECT][PYTHON] Get or create the sessions of
SparkSession.builder as PySpark does
### What changes were proposed in this pull request?
[SPARK-61008](https://issues.apache.org/jira/browse/SPARK-61008):
`SparkSession.builder` now gets or creates sessions as PySpark 4.3.0's does.
- `getOrCreate()` returns the active session of the thread, or else the
default session, as the same object, and creates one only when there is
neither. It sets the options of the builder on the session that it returns.
- The options are set in one request whose failures are ignored, as
`_apply_options` sends. That ignores, for example, static configurations.
`spark.remote` and `spark.master` are not options, and `appName()` sets
`spark.app.name`.
- The URL is that of `remote()`, or else of `SPARK_REMOTE`. Without either,
`PySparkRuntimeError` `CONNECT_URL_NOT_SET` is raised.
- The first session created is the default session, and the active session
is thread-local. `create()` makes a new session, which is the default and the
active session only when they are not set. `stop()` unsets them, and
`SparkSession.active()` returns the active or else the default session.
The core configuration has `set_all`, of several values in one request,
silently or not, for this.
### Why are the changes needed?
`getOrCreate()` created a new session each time, so the configurations and
temporary views of a session were not those of the next one. After
`spark.conf.set("spark.sql.shuffle.partitions", "7")` and
`createOrReplaceTempView("v")`, a second `getOrCreate()` returned a session
whose configuration was `200` and that had no view `v`. A library that calls
`getOrCreate()` got a session of its own. `getOrCreate()` also raised
`ValueError` without `remote()`, even with `SPARK_REMOTE` set. One [...]
### Does this PR introduce _any_ user-facing change?
Yes. `getOrCreate()`, `create()`, `getActiveSession()` and `active()`
behave as in PySpark 4.3.0, and an existing session is returned with the
options of the builder set on it.
### How was this patch tested?
- A new offline test checks, in a new interpreter:
- `getOrCreate()` returns the active session, as the same object,
whatever the URL, and the default session in another thread.
- `create()` does not replace the active session.
- `stop()` unsets the sessions.
- The URL comes from `SPARK_REMOTE`, and `CONNECT_URL_NOT_SET` and
`NO_ACTIVE_OR_DEFAULT_SESSION` are raised.
- A new offline test checks the options that a mock Connect server
receives. They arrive in one silent request, without `spark.master`, and once
for an existing session too. The mock now serves Config `Set` requests.
- A new live test checks that `getOrCreate()` returns the session with an
option set, and that a static configuration is ignored. It also passes with
official PySpark 4.3.0.
- The new tests fail without this change.
- The lib tests and every integration test, e2e included, against a local
Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of
all tests with `wasm-udf`, the offline Python suites (1161 passed, 8 skipped),
and the live drop-in and Arrow suites (87 passed) and live cases (10 passed).
Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #211 from HyukjinKwon/SPARK-61008-get-or-create.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/session.rs | 210 ++++++++++++++++++++++++--------
crates/spark-connect-core/src/client.rs | 31 +++++
crates/spark-connect/src/conf.rs | 6 +
python/tests/_mock_connect_server.py | 16 ++-
python/tests/test_dropin_e2e.py | 19 +++
python/tests/test_dropin_offline.py | 71 +++++++++++
6 files changed, 303 insertions(+), 50 deletions(-)
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 2536399..dafb3cf 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -107,13 +107,13 @@ impl PySparkSessionBuilder {
Ok(b)
}
- /// `appName` - accepted for API parity; a Connect client cannot rename an
- /// already-running remote server, so this is a no-op (chainable), matching
- /// pyspark's Connect behavior.
+ /// `appName` - the option `spark.app.name`, as PySpark's builder sets.
Like any option,
+ /// it is set on the session silently, so it has no effect on a running
server.
#[pyo3(name = "appName")]
fn app_name(&self, name: &str) -> PySparkSessionBuilder {
- let _ = name;
- self.clone()
+ let mut b = self.clone();
+ b.set_conf("spark.app.name", name.to_string());
+ b
}
/// `master` - not applicable to a remote Connect session (no local
cluster to
@@ -184,29 +184,36 @@ impl PySparkSessionBuilder {
self.clone()
}
- /// Build and get or create the session (`getOrCreate`).
+ /// Get the active session of this thread, or else the default session, or
create one
+ /// (`getOrCreate`), and set the options of the builder on it. Mirrors
PySpark's: the URL
+ /// is that of `remote()` or else of `SPARK_REMOTE`, and an existing
session is returned
+ /// whatever its URL.
#[pyo3(name = "getOrCreate")]
- fn get_or_create(&self, py: Python<'_>) -> PyResult<PySparkSession> {
- let url = self.remote_url.clone().ok_or_else(|| {
- pyo3::exceptions::PyValueError::new_err("Must call .remote(url)
before .getOrCreate()")
- })?;
-
- let builder = SparkSession::builder().remote(&url);
- let session = py.detach(|| builder.get_or_create()).to_pyerr()?;
- // Apply builder configs as runtime confs now that we have a session.
- let conf = session.conf();
- for (k, v) in &self.configs {
- py.detach(|| conf.set(k, v)).to_pyerr()?;
- }
- *active_slot().lock().unwrap() = Some(session.clone());
+ fn get_or_create(&self, py: Python<'_>) -> PyResult<Py<PySparkSession>> {
+ let url = self.url(py)?;
+ let session = {
+ let _creating = session_lock();
+ match active_session(py).or_else(|| default_session(py)) {
+ Some(session) => session,
+ None => new_session(py, &url)?,
+ }
+ };
+ self.apply_options(py, &session.borrow(py).session)?;
log_client_banner(py);
- Ok(PySparkSession::new(session))
+ Ok(session)
}
- /// `create()` - builds a brand-new session. For Connect this is
equivalent to
- /// getOrCreate against the configured remote.
- fn create(&self, py: Python<'_>) -> PyResult<PySparkSession> {
- self.get_or_create(py)
+ /// `create()` - a new session, which is the default session, and the
active one of this
+ /// thread, only when they are not set. Mirrors PySpark's.
+ fn create(&self, py: Python<'_>) -> PyResult<Py<PySparkSession>> {
+ let url = self.url(py)?;
+ let session = {
+ let _creating = session_lock();
+ new_session(py, &url)?
+ };
+ self.apply_options(py, &session.borrow(py).session)?;
+ log_client_banner(py);
+ Ok(session)
}
}
@@ -219,15 +226,115 @@ impl PySparkSessionBuilder {
self.configs.push((key.to_string(), value));
}
}
+
+ /// The URL of the session: that of `remote()`, or else of `SPARK_REMOTE`,
as PySpark's
+ /// builder of a Connect-only client, which raises `CONNECT_URL_NOT_SET`
without either.
+ fn url(&self, py: Python<'_>) -> PyResult<String> {
+ match self
+ .remote_url
+ .clone()
+ .or_else(|| std::env::var("SPARK_REMOTE").ok())
+ {
+ Some(url) => Ok(url),
+ None => Err(crate::column::raise_pyspark(
+ py,
+ "PySparkRuntimeError",
+ "CONNECT_URL_NOT_SET",
+ &[],
+ )),
+ }
+ }
+
+ /// Set the options of the builder on `session`, as PySpark's
`_apply_options`: in one
+ /// request whose failures are ignored, such as those of static
configurations, and
+ /// without `spark.master`, which a session does not take.
+ // SparkError is large, as the core crate allows for its API.
+ #[allow(clippy::result_large_err)]
+ fn apply_options(&self, py: Python<'_>, session: &SparkSession) ->
PyResult<()> {
+ let options: Vec<(String, String)> = self
+ .configs
+ .iter()
+ .filter(|(key, _)| key != "spark.master")
+ .cloned()
+ .collect();
+ if options.is_empty() {
+ return Ok(());
+ }
+ let conf = session.conf();
+ py.detach(|| conf.set_all(&options, true)).to_pyerr()
+ }
}
-/// Process-global "active" session (mirrors
`SparkSession.active`/`getActiveSession`).
-fn active_slot() -> &'static std::sync::Mutex<Option<SparkSession>> {
- static SLOT: std::sync::OnceLock<std::sync::Mutex<Option<SparkSession>>> =
+/// The default session: the first session created, of every thread, until it
stops. Mirrors
+/// `SparkSession._default_session`. The Python object, which `getOrCreate()`
returns as it is.
+fn default_slot() -> &'static std::sync::Mutex<Option<Py<PySparkSession>>> {
+ static SLOT:
std::sync::OnceLock<std::sync::Mutex<Option<Py<PySparkSession>>>> =
std::sync::OnceLock::new();
SLOT.get_or_init(|| std::sync::Mutex::new(None))
}
+thread_local! {
+ /// The active session of this thread, until it stops. Mirrors the
thread-local
+ /// `SparkSession._active_session`.
+ static ACTIVE_SESSION: std::cell::RefCell<Option<Py<PySparkSession>>> =
+ const { std::cell::RefCell::new(None) };
+}
+
+/// Held while a session is got or created, as PySpark's `SparkSession._lock`,
so that threads
+/// get the same default session. Taken with the GIL, which is not released
while it is held.
+fn session_lock() -> std::sync::MutexGuard<'static, ()> {
+ static LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
+ LOCK.lock().unwrap_or_else(|e| e.into_inner())
+}
+
+/// `session`, unless it stopped.
+fn unless_stopped(py: Python<'_>, session: &Py<PySparkSession>) ->
Option<Py<PySparkSession>> {
+ (!session.borrow(py).session.is_stopped()).then(|| session.clone_ref(py))
+}
+
+/// The active session of this thread, unless it stopped.
+fn active_session(py: Python<'_>) -> Option<Py<PySparkSession>> {
+ ACTIVE_SESSION.with(|active| {
+ active
+ .borrow()
+ .as_ref()
+ .and_then(|session| unless_stopped(py, session))
+ })
+}
+
+/// The default session, unless it stopped.
+fn default_session(py: Python<'_>) -> Option<Py<PySparkSession>> {
+ default_slot()
+ .lock()
+ .unwrap_or_else(|e| e.into_inner())
+ .as_ref()
+ .and_then(|session| unless_stopped(py, session))
+}
+
+/// A new session of `url`, which is the default session, and the active
session of this
+/// thread, when they are not set, as PySpark's
`_set_default_and_active_session`.
+fn new_session(py: Python<'_>, url: &str) -> PyResult<Py<PySparkSession>> {
+ // The channel connects lazily, so this does not wait on the server.
+ let session = SparkSession::builder()
+ .remote(url)
+ .get_or_create()
+ .to_pyerr()?;
+ let session = Py::new(py, PySparkSession::new(session))?;
+ {
+ let mut default = default_slot().lock().unwrap_or_else(|e|
e.into_inner());
+ if default.is_none() {
+ *default = Some(session.clone_ref(py));
+ }
+ }
+ ACTIVE_SESSION.with(|active| {
+ let mut active = active.borrow_mut();
+ if active.is_none() {
+ *active = Some(session.clone_ref(py));
+ }
+ });
+ Ok(session)
+}
+
/// Emit a one-time INFO log identifying this as the Rust-backed Spark Connect
/// client, so it is discoverable at runtime which client is in use -- and
issues
/// get filed against this project, not the reference `pyspark-client`. Logged
the
@@ -996,13 +1103,22 @@ impl PySparkSession {
),),
)?;
}
+ // As PySpark's, a stopped session is no longer the default session,
or the active
+ // session of this thread.
let sid = self.session.session_id();
+ let is_this = |s: &Py<PySparkSession>|
s.borrow(py).session.session_id() == sid;
{
- let mut slot = active_slot().lock().unwrap();
- if slot.as_ref().is_some_and(|s| s.session_id() == sid) {
- *slot = None;
+ let mut default = default_slot().lock().unwrap_or_else(|e|
e.into_inner());
+ if default.as_ref().is_some_and(is_this) {
+ *default = None;
}
}
+ ACTIVE_SESSION.with(|active| {
+ let mut active = active.borrow_mut();
+ if active.as_ref().is_some_and(is_this) {
+ *active = None;
+ }
+ });
// Only touch the SQLContext cache if the module was ever imported (as
pyspark does).
let modules = py.import("sys")?.getattr("modules")?;
if let Ok(ctx_mod) = modules.get_item("pyspark.sql.connect.context") {
@@ -1139,29 +1255,27 @@ impl PySparkSession {
.call0()
}
- /// The active session for this process, or None. Mirrors
`SparkSession.getActiveSession`.
+ /// The active session of this thread, or None. Mirrors
`SparkSession.getActiveSession`.
#[staticmethod]
#[pyo3(name = "getActiveSession")]
- fn get_active_session() -> Option<PySparkSession> {
- active_slot()
- .lock()
- .unwrap()
- .clone()
- .map(PySparkSession::new)
+ fn get_active_session(py: Python<'_>) -> Option<Py<PySparkSession>> {
+ active_session(py)
}
- /// The active session; errors if none. Mirrors `SparkSession.active`,
which raises
- /// `NO_ACTIVE_OR_DEFAULT_SESSION`.
+ /// The active session of this thread, or else the default session; errors
if neither.
+ /// Mirrors `SparkSession.active`, which raises
`NO_ACTIVE_OR_DEFAULT_SESSION`.
#[staticmethod]
- fn active(py: Python<'_>) -> PyResult<PySparkSession> {
- Self::get_active_session().ok_or_else(|| {
- crate::column::raise_pyspark(
- py,
- "PySparkRuntimeError",
- "NO_ACTIVE_OR_DEFAULT_SESSION",
- &[],
- )
- })
+ fn active(py: Python<'_>) -> PyResult<Py<PySparkSession>> {
+ active_session(py)
+ .or_else(|| default_session(py))
+ .ok_or_else(|| {
+ crate::column::raise_pyspark(
+ py,
+ "PySparkRuntimeError",
+ "NO_ACTIVE_OR_DEFAULT_SESSION",
+ &[],
+ )
+ })
}
/// Register a progress handler invoked on each execution-progress message.
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 1c1881a..09ca9c4 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -701,6 +701,37 @@ impl SparkConnectClient {
Ok(())
}
+ /// Set configuration values in one request, ignoring the failures to set
them when
+ /// `silent`. Mirrors `pyspark.sql.connect.conf.RuntimeConf._set_all`,
with which the
+ /// session builder applies its options.
+ pub async fn set_configs(&self, pairs: &[(String, String)], silent: bool)
-> Result<()> {
+ let set = spark_connect_proto::config_request::Set {
+ pairs: pairs
+ .iter()
+ .map(|(key, value)| spark_connect_proto::KeyValue {
+ key: key.clone(),
+ value: Some(value.clone()),
+ })
+ .collect(),
+ silent: Some(silent),
+ };
+ let request = ConfigRequest {
+ session_id: self.session_id.clone(),
+ user_context: Some(UserContext {
+ user_id: self.user_id.clone().unwrap_or_default(),
+ ..Default::default()
+ }),
+ operation: Some(spark_connect_proto::config_request::Operation {
+ op_type:
Some(spark_connect_proto::config_request::operation::OpType::Set(
+ set,
+ )),
+ }),
+ ..Default::default()
+ };
+ let _ = self.config(request).await?;
+ Ok(())
+ }
+
/// Get configuration values with default fallbacks.
///
/// Mirrors
`pyspark.sql.connect.client.core.SparkConnectClient.get_config_with_defaults`.
diff --git a/crates/spark-connect/src/conf.rs b/crates/spark-connect/src/conf.rs
index 3227f11..10d38c1 100644
--- a/crates/spark-connect/src/conf.rs
+++ b/crates/spark-connect/src/conf.rs
@@ -43,6 +43,12 @@ impl RuntimeConf {
block_on(self.client.set_config(key, value))
}
+ /// Set several configuration values in one request, ignoring the failures
to set them,
+ /// such as of static configurations, when `silent`. Mirrors
`RuntimeConf._set_all`.
+ pub fn set_all(&self, pairs: &[(String, String)], silent: bool) ->
Result<()> {
+ block_on(self.client.set_configs(pairs, silent))
+ }
+
/// Get a configuration value by key.
///
/// Returns `Ok(None)` if the key does not exist on the server.
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index c2ea1eb..53a5e6c 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -125,6 +125,7 @@ def serve(connection):
self.fetched_error_ids = []
self.confs = {}
self.read_confs = []
+ self.set_confs = []
def analyze(self, request, context):
self.calls += 1
@@ -160,7 +161,8 @@ def serve(connection):
# FetchErrorDetails returns `error_details` (raw
FetchErrorDetailsResponse fields) and
# records the error id asked for. Config Get returns the values in
`confs`, or else
# those of `default_confs`, the defaults of SQL configs of the server,
and fails for a
- # key without one, as the server does for an unset config; it records
the keys.
+ # key without one, as the server does for an unset config; it records
the keys. Config
+ # Set records the pairs and the silent flag of each request.
error_details = b""
fetched_error_ids = []
confs = {}
@@ -169,6 +171,7 @@ def serve(connection):
"spark.sql.execution.pythonUDTF.arrow.enabled": "true",
}
read_confs = []
+ set_confs = []
def execute(self, request, context):
self.execute_calls += 1
@@ -220,8 +223,17 @@ def serve(connection):
# ConfigResponse.session_id=1, pairs=2, KeyValue.key=1, value=2.
request = parse_message(request)
operation = parse_message(request[3][0])
+ if 1 in operation:
+ # Operation.set=1: Set.pairs=1 (KeyValue.key=1, value=2),
silent=2.
+ set_request = parse_message(operation[1][0])
+ pairs = []
+ for pair in set_request.get(1, []):
+ pair = parse_message(pair)
+ pairs.append((pair[1][0].decode(), pair[2][0].decode()))
+ self.set_confs.append((pairs, set_request.get(2, [None])[0]))
+ return message_field(1, request[1][0])
if 2 not in operation:
- context.abort(grpc.StatusCode.UNIMPLEMENTED, "only Get is
served")
+ context.abort(grpc.StatusCode.UNIMPLEMENTED, "only Get and Set
are served")
keys = [key.decode() for key in
parse_message(operation[2][0]).get(1, [])]
self.read_confs.extend(keys)
confs = {**self.default_confs, **self.confs}
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index e604d25..1472a4b 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -74,6 +74,25 @@ def test_conf_get_with_a_none_default(session):
session.conf.get(key)
+def
test_get_or_create_returns_the_session_with_the_options_as_pyspark(session):
+ # As PySpark's builder: getOrCreate() returns the active session, and sets
the options
+ # of the builder on it; one that fails, of a static configuration, is
ignored.
+ key = "spark.sql.shuffle.partitions"
+ previous = session.conf.get(key)
+ try:
+ got = (
+ SparkSession.builder.remote(os.environ["SPARK_REMOTE"])
+ .config(key, "5")
+ .config("spark.sql.warehouse.dir", "/not/set")
+ .getOrCreate()
+ )
+ assert got is session
+ assert session.conf.get(key) == "5"
+ assert session.conf.get("spark.sql.warehouse.dir") != "/not/set"
+ finally:
+ session.conf.set(key, previous)
+
+
def test_repr_shows_the_rows_with_eager_evaluation_as_pyspark(session):
df = session.range(3)
assert repr(df) == "DataFrame[id: bigint]"
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 65fb374..32b52f7 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -378,6 +378,77 @@ def test_session_builder_chaining_offline():
assert type(b2).__name__ == "SparkSessionBuilder"
+def test_sessions_are_got_or_created_as_pyspark():
+ # As PySpark's builder: getOrCreate() returns the active session of the
thread, or else
+ # the default session, whatever the URL, and creates one when there is
neither, of the
+ # URL of remote() or else of SPARK_REMOTE; create() makes a new one, which
is the default
+ # and the active session only when they are not set; and stop() unsets
them. In a new
+ # interpreter, which has no default session.
+ import subprocess
+ import sys
+
+ subprocess.run([sys.executable, "-c", """
+import os
+import threading
+
+from pyspark.errors import PySparkRuntimeError
+from pyspark.sql import SparkSession
+
+def condition(call):
+ try:
+ call()
+ except PySparkRuntimeError as error:
+ return error.getCondition()
+ raise AssertionError("no error")
+
+os.environ.pop("SPARK_REMOTE", None)
+assert condition(SparkSession.builder.getOrCreate) == "CONNECT_URL_NOT_SET"
+assert condition(SparkSession.active) == "NO_ACTIVE_OR_DEFAULT_SESSION"
+
+first = SparkSession.builder.remote("sc://localhost:1").getOrCreate()
+assert SparkSession.builder.remote("sc://localhost:2").getOrCreate() is first
+assert SparkSession.getActiveSession() is first and SparkSession.active() is
first
+
+seen = {}
+def other_thread():
+ seen["active"] = SparkSession.getActiveSession()
+ seen["default"] = SparkSession.active()
+ seen["got"] = SparkSession.builder.remote("sc://localhost:1").getOrCreate()
+thread = threading.Thread(target=other_thread)
+thread.start()
+thread.join()
+assert seen == {"active": None, "default": first, "got": first}, seen
+
+created = SparkSession.builder.remote("sc://localhost:1").create()
+assert created is not first and SparkSession.getActiveSession() is first
+
+first.stop()
+assert SparkSession.getActiveSession() is None
+assert condition(SparkSession.active) == "NO_ACTIVE_OR_DEFAULT_SESSION"
+os.environ["SPARK_REMOTE"] = "sc://localhost:1"
+again = SparkSession.builder.getOrCreate()
+assert again is not first and again is not created
+assert SparkSession.getActiveSession() is again and SparkSession.active() is
again
+"""], check=True)
+
+
+def test_builder_options_are_set_on_the_session_as_pyspark(mock_connect):
+ # As PySpark's _apply_options: the options of the builder, but
spark.remote and
+ # spark.master, are set on the session that getOrCreate() returns, also an
existing one,
+ # in one request whose failures are ignored.
+ from pyspark.sql import SparkSession
+
+ spark, service = mock_connect
+ builder =
SparkSession.builder.remote("sc://localhost:1").master("local[4]")
+ builder = builder.appName("app").config("spark.sql.shuffle.partitions", 3)
+ assert builder.getOrCreate() is spark
+ assert service.set_confs == [
+ ([("spark.app.name", "app"), ("spark.sql.shuffle.partitions", "3")], 1)
+ ]
+ assert SparkSession.builder.remote("sc://localhost:1").getOrCreate() is
spark
+ assert len(service.set_confs) == 1
+
+
def test_session_builder_class_constructor():
from pyspark.sql import SparkSession
b = SparkSession.Builder()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]