This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 64b63268 [SPARK-61183][CONNECT][PYTHON] Give back the SparkSession
object of a DataFrame in DataFrame.sparkSession, in the Python client as
PySpark does
64b63268 is described below
commit 64b6326827229f17aa7bb1bcfe7bac7d75aeae64
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 06:21:53 2026 +0900
[SPARK-61183][CONNECT][PYTHON] Give back the SparkSession object of a
DataFrame in DataFrame.sparkSession, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61183](https://issues.apache.org/jira/browse/SPARK-61183): PySpark
gives back the session object itself from `DataFrame.sparkSession` and
`DataFrame._session`, so `df.sparkSession is spark` holds for the DataFrames of
a session, and so do `df.sparkSession.streams is spark.streams` and
`df.sparkSession.client is spark.client`.
- The session objects are registered by session ID in a registry that holds
them as weak references (`weakref.ref`). A getter that gives back a session
takes the live object of its session from the registry, and makes and registers
a new one only when none is alive. Sessions made apart may share a session ID
(two of an explicit `session_id`), so the registry keeps a list per ID and
compares the core sessions with the new `SparkSession::is_same_session` of the
Rust client (a session an [...]
- `newSession` and `cloneSession` now return the registered object, not a
new one.
- `DataSourceRegistration.sparkSession`: the session of the registration,
which PySpark has and the client lacked.
### Why are the changes needed?
Each access made a new session object, so the identity of the session was
lost, along with the `streams` and the `client` that are cached per object:
```python
spark.range(2).sparkSession is spark
# PySpark 4.3.0: True
# before this change: False
spark.range(2).sparkSession.streams is spark.streams
# PySpark 4.3.0: True
# before this change: False
spark.range(2).sparkSession.client is spark.client
# PySpark 4.3.0: True
# before this change: False
spark.dataSource.sparkSession
# PySpark 4.3.0: the session
# before this change: AttributeError
```
### Does this PR introduce _any_ user-facing change?
Yes. `DataFrame.sparkSession` and `DataFrame._session` return the session
object of the DataFrame, the same object as `spark` for its session, and so do
the `streams` and `client` of it. `newSession` and `cloneSession` return the
object of the new session, which is registered until it is dropped.
`DataSourceRegistration.sparkSession` is added.
### How was this patch tested?
- `test_dataframes_give_back_their_session_object_as_pyspark` in
`python/tests/test_dropin_offline.py`. It checks the identity of the session
for `range`, `select`, `groupBy().count()`, `na.drop`, `read`, `udf`, `udtf`
and `dataSource`, the identity of `streams` and `client`, that two sessions do
not mix (their objects and streams differ), and that a dropped `newSession`
object is freed (`weakref` and `gc`), while the DataFrames of `spark` keep
giving back `spark`, and, in a process o [...]
- It uses a `check` helper that raises, not an `assert`: pytest reprs the
operands of a failed `assert`, and reprs of DataFrames ask the server for their
schema, which hung the baseline run.
- The lib tests of both crates (608 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2228 passed, 8 skipped), and the
live drop-in and Arrow suites (210 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #378 from HyukjinKwon/SPARK-61183-dataframe-session-identity.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/catalog.rs | 4 +-
crates/pyspark-rs/src/dataframe.rs | 10 ++---
crates/pyspark-rs/src/datasource.rs | 13 ++++--
crates/pyspark-rs/src/session.rs | 89 ++++++++++++++++++++++++++++++++++---
crates/pyspark-rs/src/streaming.rs | 2 +-
crates/spark-connect/src/session.rs | 6 +++
python/tests/test_dropin_offline.py | 55 +++++++++++++++++++++++
7 files changed, 161 insertions(+), 18 deletions(-)
diff --git a/crates/pyspark-rs/src/catalog.rs b/crates/pyspark-rs/src/catalog.rs
index d0ca8da4..e6a6a45a 100644
--- a/crates/pyspark-rs/src/catalog.rs
+++ b/crates/pyspark-rs/src/catalog.rs
@@ -496,8 +496,8 @@ impl PyCatalog {
future_warning,
),
)?;
- let session =
crate::session::PySparkSession::new(self.catalog.spark_session().clone());
- Py::new(py, session)?
+ let session = crate::session::session_object(py,
self.catalog.spark_session())?;
+ session
.bind(py)
.getattr("udf")?
.call_method1("register", (name, f, returnType))
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 4b4d9d0f..272494d2 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -670,7 +670,7 @@ impl PyDataFrame {
let this = slf.borrow();
py.import("pyspark.sql.connect.dataframe")?
.getattr("_reduce_dataframe")?
- .call1((this.spark_session(), this._relation_bytes(py)))
+ .call1((this.spark_session(py)?, this._relation_bytes(py)))
}
/// The DataFrame of `relation`, the bytes of a relation of `session`, as
a pickled DataFrame
@@ -1973,15 +1973,15 @@ impl PyDataFrame {
/// The SparkSession that created this DataFrame.
#[pyo3(name = "sparkSession")]
#[getter]
- fn spark_session(&self) -> crate::session::PySparkSession {
- crate::session::PySparkSession::new(self.dataframe.spark_session())
+ fn spark_session(&self, py: Python<'_>) ->
PyResult<Py<crate::session::PySparkSession>> {
+ crate::session::session_object(py, &self.dataframe.spark_session())
}
/// The SparkSession of this DataFrame, as PySpark Connect's
`DataFrame._session`, which the
/// modules of PySpark read, such as `pyspark.ml`, `pyspark.sql.plot` and
`pyspark.pandas`.
#[getter]
- fn _session(&self) -> crate::session::PySparkSession {
- self.spark_session()
+ fn _session(&self, py: Python<'_>) ->
PyResult<Py<crate::session::PySparkSession>> {
+ self.spark_session(py)
}
/// Not supported on Spark Connect, as PySpark Connect's:
`NOT_IMPLEMENTED` of `rdd`, which
diff --git a/crates/pyspark-rs/src/datasource.rs
b/crates/pyspark-rs/src/datasource.rs
index 521d921c..20aa3849 100644
--- a/crates/pyspark-rs/src/datasource.rs
+++ b/crates/pyspark-rs/src/datasource.rs
@@ -19,17 +19,23 @@ use crate::session::PySparkSession;
dict
)]
pub struct PyDataSourceRegistration {
- pub(crate) session: PySparkSession,
+ pub(crate) session: Py<PySparkSession>,
}
impl PyDataSourceRegistration {
- pub fn new(session: PySparkSession) -> Self {
+ pub fn new(session: Py<PySparkSession>) -> Self {
PyDataSourceRegistration { session }
}
}
#[pymethods]
impl PyDataSourceRegistration {
+ /// The session of the registration, as PySpark's
`DataSourceRegistration.sparkSession`.
+ #[getter(sparkSession)]
+ fn spark_session(&self, py: Python<'_>) -> Py<PySparkSession> {
+ self.session.clone_ref(py)
+ }
+
/// Register a custom data source class.
///
/// Mirrors `pyspark.sql.SparkSession.dataSource.register`.
@@ -48,7 +54,8 @@ impl PyDataSourceRegistration {
let expr = CommonInlineUserDefinedDataSourceExpression::new(name,
payload);
// Register with the session (release GIL during the RPC)
- py.detach(|| self.session.session.register_data_source(expr))
+ let session = self.session.borrow(py).session.clone();
+ py.detach(|| session.register_data_source(expr))
.to_pyerr()?;
Ok(())
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 773514e7..6a42d5dc 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -470,11 +470,83 @@ fn new_session(py: Python<'_>, url: &str) ->
PyResult<Py<PySparkSession>> {
.remote(url)
.get_or_create()
.to_pyerr()?;
- let session = Py::new(py, PySparkSession::new(session))?;
+ let session = session_object(py, &session)?;
set_default_and_active(py, &session);
Ok(session)
}
+/// The Python session objects by session ID, held weakly, so that the
registry keeps none
+/// alive. Sessions made apart may share a session ID, such as two of an
explicit `session_id`,
+/// so an ID has a list of them.
+type SessionObjects = std::collections::HashMap<String, Vec<Py<PyAny>>>;
+
+/// The registry of the Python session objects.
+fn session_objects() -> &'static std::sync::Mutex<SessionObjects> {
+ static OBJECTS: std::sync::OnceLock<std::sync::Mutex<SessionObjects>> =
+ std::sync::OnceLock::new();
+ OBJECTS.get_or_init(|| std::sync::Mutex::new(SessionObjects::new()))
+}
+
+/// The session object that the weak reference `weak` refers to, unless it was
dropped.
+fn live_session_object(py: Python<'_>, weak: &Py<PyAny>) ->
Option<Py<PySparkSession>> {
+ let object = weak.bind(py).call0().ok()?;
+ let session = object.cast::<PySparkSession>().ok()?;
+ Some(session.clone().unbind())
+}
+
+/// The live object of `session` itself in `registry`, if any, and not that of
another session
+/// of its session ID.
+fn registered_session_object(
+ py: Python<'_>,
+ registry: &SessionObjects,
+ session: &SparkSession,
+) -> Option<Py<PySparkSession>> {
+ registry
+ .get(session.session_id())?
+ .iter()
+ .filter_map(|weak| live_session_object(py, weak))
+ .find(|object| {
+ object
+ .bind(py)
+ .try_borrow()
+ .is_ok_and(|object| object.session.is_same_session(session))
+ })
+}
+
+/// The Python object of `session`, one per session as PySpark has them: its
live object, or
+/// else a new one, which the registry holds weakly. So the DataFrames, the
streams and the
+/// clients of a session give back the session's object, as PySpark's do.
+pub(crate) fn session_object(
+ py: Python<'_>,
+ session: &SparkSession,
+) -> PyResult<Py<PySparkSession>> {
+ {
+ let registry = session_objects().lock().unwrap_or_else(|e|
e.into_inner());
+ if let Some(object) = registered_session_object(py, ®istry,
session) {
+ return Ok(object);
+ }
+ }
+ let object = Py::new(py, PySparkSession::new(session.clone()))?;
+ let weak = py
+ .import("weakref")?
+ .call_method1("ref", (&object,))?
+ .unbind();
+ let mut registry = session_objects().lock().unwrap_or_else(|e|
e.into_inner());
+ // Another thread may have made the object of this session meanwhile; then
it is that one.
+ if let Some(existing) = registered_session_object(py, ®istry, session) {
+ return Ok(existing);
+ }
+ registry.retain(|_, objects| {
+ objects.retain(|weak| live_session_object(py, weak).is_some());
+ !objects.is_empty()
+ });
+ registry
+ .entry(session.session_id().to_string())
+ .or_default()
+ .push(weak);
+ Ok(object)
+}
+
/// Make `session` the default session, and the active session of this thread,
when they are
/// not set, as PySpark's `SparkSession._set_default_and_active_session`.
fn set_default_and_active(py: Python<'_>, session: &Py<PySparkSession>) {
@@ -2253,8 +2325,11 @@ impl PySparkSession {
///
/// Mirrors `SparkSession.dataSource.register`. Used to register custom
data sources.
#[getter]
- fn dataSource(&self) -> PyDataSourceRegistration {
-
PyDataSourceRegistration::new(PySparkSession::new(self.session.clone()))
+ fn dataSource(&self, py: Python<'_>) -> PyResult<PyDataSourceRegistration>
{
+ Ok(PyDataSourceRegistration::new(session_object(
+ py,
+ &self.session,
+ )?))
}
/// Build and register a resource profile with the server.
@@ -2443,8 +2518,8 @@ impl PySparkSession {
}
#[pyo3(name = "newSession")]
- fn new_session(&self) -> PySparkSession {
- PySparkSession::new(self.session.new_session())
+ fn new_session(&self, py: Python<'_>) -> PyResult<Py<PySparkSession>> {
+ session_object(py, &self.session.new_session())
}
/// Clone this session on the server, as PySpark's
`cloneSession(new_session_id=None)`: a
@@ -2456,7 +2531,7 @@ impl PySparkSession {
&self,
py: Python<'_>,
new_session_id: Option<String>,
- ) -> PyResult<PySparkSession> {
+ ) -> PyResult<Py<PySparkSession>> {
let session = py
.detach(|| {
self.session
@@ -2464,7 +2539,7 @@ impl PySparkSession {
.map_err(Box::new)
})
.to_pyerr()?;
- Ok(PySparkSession::new(session))
+ session_object(py, &session)
}
}
diff --git a/crates/pyspark-rs/src/streaming.rs
b/crates/pyspark-rs/src/streaming.rs
index 58a3c9c4..8376be3b 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -1245,7 +1245,7 @@ impl PyStreamingQueryManager {
/// listener added twice is in the bus twice. Should the server fail to
add its listener, this
/// warns, as PySpark Connect's does.
fn addListener(&self, py: Python<'_>, listener: Py<PyAny>) -> PyResult<()>
{
- let session =
crate::session::PySparkSession::new(self.session.clone());
+ let session = crate::session::session_object(py, &self.session)?;
listener
.bind(py)
.call_method1("_set_spark_session", (session,))?;
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 6dc8d152..de616a79 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -394,6 +394,12 @@ impl SparkSession {
&self.client
}
+ /// Whether `other` is this session or a clone of it, which shares its
client. Two sessions
+ /// made apart are not the same session, even of the same session ID.
+ pub fn is_same_session(&self, other: &SparkSession) -> bool {
+ Arc::ptr_eq(&self.client, &other.client)
+ }
+
/// Create a builder for a new SparkSession.
pub fn builder() -> SparkSessionBuilder {
SparkSessionBuilder::new()
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 252cd1ae..36900270 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1068,6 +1068,61 @@ def
test_session_builders_set_their_options_and_validate_their_urls_as_pyspark(m
+def test_dataframes_give_back_their_session_object_as_pyspark(offline_spark):
+ # As PySpark: the DataFrames of a session, its streams and its client give
back the session
+ # object itself, and two sessions do not mix. The registry holds the
session objects
+ # weakly, so a session that is dropped is freed.
+ import gc
+ import weakref
+
+ def check(actual, expected):
+ # Not an assert: pytest reprs a DataFrame in a failed assert, which
asks the server.
+ if actual is not expected:
+ raise AssertionError("not the session object that PySpark gives
back")
+
+ spark = offline_spark
+ df = spark.range(2)
+ check(df.sparkSession, spark)
+ check(df._session, spark)
+ check(df.select("id").sparkSession, spark)
+ check(df.groupBy("id").count().sparkSession, spark)
+ check(df.na.drop().sparkSession, spark)
+ check(spark.read.format("rate").load().sparkSession, spark)
+ check(spark.udf.sparkSession, spark)
+ check(spark.udtf.sparkSession, spark)
+ check(spark.dataSource.sparkSession, spark)
+ assert df.sparkSession == spark
+ check(df.sparkSession.streams, spark.streams)
+ check(df.sparkSession.client, spark.client)
+ other = spark.newSession()
+ check(other.range(1).sparkSession, other)
+ if other.range(1).sparkSession is spark:
+ raise AssertionError("two sessions mixed")
+ check(other.range(1).sparkSession.streams, other.streams)
+ if other.streams is spark.streams:
+ raise AssertionError("two sessions share their streams")
+ ref = weakref.ref(other)
+ del other
+ gc.collect()
+ assert ref() is None
+ check(df.sparkSession, spark)
+ # Two sessions made apart, of the same session ID, are two objects, each
that of its own
+ # DataFrames. create() sets the default session, so this runs in a process
of its own.
+ import subprocess
+ import sys
+
+ subprocess.run([sys.executable, "-c", """
+import uuid
+from pyspark.sql import SparkSession
+url = f"sc://localhost:1/;session_id={uuid.uuid4()}"
+first = SparkSession.builder.remote(url).create()
+second = SparkSession.builder.remote(url).create()
+assert first is not second
+assert first.range(1).sparkSession is first
+assert second.range(1).sparkSession is second
+"""], check=True)
+
+
def test_session_builder_class_constructor():
from pyspark.sql import SparkSession
b = SparkSession.Builder()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]