This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 64b63268 [SPARK-61183][CONNECT][PYTHON] Give back the SparkSession 
object of a DataFrame in DataFrame.sparkSession, in the Python client as 
PySpark does
64b63268 is described below

commit 64b6326827229f17aa7bb1bcfe7bac7d75aeae64
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 06:21:53 2026 +0900

    [SPARK-61183][CONNECT][PYTHON] Give back the SparkSession object of a 
DataFrame in DataFrame.sparkSession, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61183](https://issues.apache.org/jira/browse/SPARK-61183): PySpark 
gives back the session object itself from `DataFrame.sparkSession` and 
`DataFrame._session`, so `df.sparkSession is spark` holds for the DataFrames of 
a session, and so do `df.sparkSession.streams is spark.streams` and 
`df.sparkSession.client is spark.client`.
    
    - The session objects are registered by session ID in a registry that holds 
them as weak references (`weakref.ref`). A getter that gives back a session 
takes the live object of its session from the registry, and makes and registers 
a new one only when none is alive. Sessions made apart may share a session ID 
(two of an explicit `session_id`), so the registry keeps a list per ID and 
compares the core sessions with the new `SparkSession::is_same_session` of the 
Rust client (a session an [...]
    - `newSession` and `cloneSession` now return the registered object, not a 
new one.
    - `DataSourceRegistration.sparkSession`: the session of the registration, 
which PySpark has and the client lacked.
    
    ### Why are the changes needed?
    
    Each access made a new session object, so the identity of the session was 
lost, along with the `streams` and the `client` that are cached per object:
    
    ```python
    spark.range(2).sparkSession is spark
    # PySpark 4.3.0: True
    # before this change: False
    spark.range(2).sparkSession.streams is spark.streams
    # PySpark 4.3.0: True
    # before this change: False
    spark.range(2).sparkSession.client is spark.client
    # PySpark 4.3.0: True
    # before this change: False
    spark.dataSource.sparkSession
    # PySpark 4.3.0: the session
    # before this change: AttributeError
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `DataFrame.sparkSession` and `DataFrame._session` return the session 
object of the DataFrame, the same object as `spark` for its session, and so do 
the `streams` and `client` of it. `newSession` and `cloneSession` return the 
object of the new session, which is registered until it is dropped. 
`DataSourceRegistration.sparkSession` is added.
    
    ### How was this patch tested?
    
    - `test_dataframes_give_back_their_session_object_as_pyspark` in 
`python/tests/test_dropin_offline.py`. It checks the identity of the session 
for `range`, `select`, `groupBy().count()`, `na.drop`, `read`, `udf`, `udtf` 
and `dataSource`, the identity of `streams` and `client`, that two sessions do 
not mix (their objects and streams differ), and that a dropped `newSession` 
object is freed (`weakref` and `gc`), while the DataFrames of `spark` keep 
giving back `spark`, and, in a process o [...]
    - It uses a `check` helper that raises, not an `assert`: pytest reprs the 
operands of a failed `assert`, and reprs of DataFrames ask the server for their 
schema, which hung the baseline run.
    - The lib tests of both crates (608 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2228 passed, 8 skipped), and the 
live drop-in and Arrow suites (210 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #378 from HyukjinKwon/SPARK-61183-dataframe-session-identity.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/catalog.rs    |  4 +-
 crates/pyspark-rs/src/dataframe.rs  | 10 ++---
 crates/pyspark-rs/src/datasource.rs | 13 ++++--
 crates/pyspark-rs/src/session.rs    | 89 ++++++++++++++++++++++++++++++++++---
 crates/pyspark-rs/src/streaming.rs  |  2 +-
 crates/spark-connect/src/session.rs |  6 +++
 python/tests/test_dropin_offline.py | 55 +++++++++++++++++++++++
 7 files changed, 161 insertions(+), 18 deletions(-)

diff --git a/crates/pyspark-rs/src/catalog.rs b/crates/pyspark-rs/src/catalog.rs
index d0ca8da4..e6a6a45a 100644
--- a/crates/pyspark-rs/src/catalog.rs
+++ b/crates/pyspark-rs/src/catalog.rs
@@ -496,8 +496,8 @@ impl PyCatalog {
                 future_warning,
             ),
         )?;
-        let session = 
crate::session::PySparkSession::new(self.catalog.spark_session().clone());
-        Py::new(py, session)?
+        let session = crate::session::session_object(py, 
self.catalog.spark_session())?;
+        session
             .bind(py)
             .getattr("udf")?
             .call_method1("register", (name, f, returnType))
diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 4b4d9d0f..272494d2 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -670,7 +670,7 @@ impl PyDataFrame {
         let this = slf.borrow();
         py.import("pyspark.sql.connect.dataframe")?
             .getattr("_reduce_dataframe")?
-            .call1((this.spark_session(), this._relation_bytes(py)))
+            .call1((this.spark_session(py)?, this._relation_bytes(py)))
     }
 
     /// The DataFrame of `relation`, the bytes of a relation of `session`, as 
a pickled DataFrame
@@ -1973,15 +1973,15 @@ impl PyDataFrame {
     /// The SparkSession that created this DataFrame.
     #[pyo3(name = "sparkSession")]
     #[getter]
-    fn spark_session(&self) -> crate::session::PySparkSession {
-        crate::session::PySparkSession::new(self.dataframe.spark_session())
+    fn spark_session(&self, py: Python<'_>) -> 
PyResult<Py<crate::session::PySparkSession>> {
+        crate::session::session_object(py, &self.dataframe.spark_session())
     }
 
     /// The SparkSession of this DataFrame, as PySpark Connect's 
`DataFrame._session`, which the
     /// modules of PySpark read, such as `pyspark.ml`, `pyspark.sql.plot` and 
`pyspark.pandas`.
     #[getter]
-    fn _session(&self) -> crate::session::PySparkSession {
-        self.spark_session()
+    fn _session(&self, py: Python<'_>) -> 
PyResult<Py<crate::session::PySparkSession>> {
+        self.spark_session(py)
     }
 
     /// Not supported on Spark Connect, as PySpark Connect's: 
`NOT_IMPLEMENTED` of `rdd`, which
diff --git a/crates/pyspark-rs/src/datasource.rs 
b/crates/pyspark-rs/src/datasource.rs
index 521d921c..20aa3849 100644
--- a/crates/pyspark-rs/src/datasource.rs
+++ b/crates/pyspark-rs/src/datasource.rs
@@ -19,17 +19,23 @@ use crate::session::PySparkSession;
     dict
 )]
 pub struct PyDataSourceRegistration {
-    pub(crate) session: PySparkSession,
+    pub(crate) session: Py<PySparkSession>,
 }
 
 impl PyDataSourceRegistration {
-    pub fn new(session: PySparkSession) -> Self {
+    pub fn new(session: Py<PySparkSession>) -> Self {
         PyDataSourceRegistration { session }
     }
 }
 
 #[pymethods]
 impl PyDataSourceRegistration {
+    /// The session of the registration, as PySpark's 
`DataSourceRegistration.sparkSession`.
+    #[getter(sparkSession)]
+    fn spark_session(&self, py: Python<'_>) -> Py<PySparkSession> {
+        self.session.clone_ref(py)
+    }
+
     /// Register a custom data source class.
     ///
     /// Mirrors `pyspark.sql.SparkSession.dataSource.register`.
@@ -48,7 +54,8 @@ impl PyDataSourceRegistration {
         let expr = CommonInlineUserDefinedDataSourceExpression::new(name, 
payload);
 
         // Register with the session (release GIL during the RPC)
-        py.detach(|| self.session.session.register_data_source(expr))
+        let session = self.session.borrow(py).session.clone();
+        py.detach(|| session.register_data_source(expr))
             .to_pyerr()?;
 
         Ok(())
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 773514e7..6a42d5dc 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -470,11 +470,83 @@ fn new_session(py: Python<'_>, url: &str) -> 
PyResult<Py<PySparkSession>> {
         .remote(url)
         .get_or_create()
         .to_pyerr()?;
-    let session = Py::new(py, PySparkSession::new(session))?;
+    let session = session_object(py, &session)?;
     set_default_and_active(py, &session);
     Ok(session)
 }
 
+/// The Python session objects by session ID, held weakly, so that the 
registry keeps none
+/// alive. Sessions made apart may share a session ID, such as two of an 
explicit `session_id`,
+/// so an ID has a list of them.
+type SessionObjects = std::collections::HashMap<String, Vec<Py<PyAny>>>;
+
+/// The registry of the Python session objects.
+fn session_objects() -> &'static std::sync::Mutex<SessionObjects> {
+    static OBJECTS: std::sync::OnceLock<std::sync::Mutex<SessionObjects>> =
+        std::sync::OnceLock::new();
+    OBJECTS.get_or_init(|| std::sync::Mutex::new(SessionObjects::new()))
+}
+
+/// The session object that the weak reference `weak` refers to, unless it was 
dropped.
+fn live_session_object(py: Python<'_>, weak: &Py<PyAny>) -> 
Option<Py<PySparkSession>> {
+    let object = weak.bind(py).call0().ok()?;
+    let session = object.cast::<PySparkSession>().ok()?;
+    Some(session.clone().unbind())
+}
+
+/// The live object of `session` itself in `registry`, if any, and not that of 
another session
+/// of its session ID.
+fn registered_session_object(
+    py: Python<'_>,
+    registry: &SessionObjects,
+    session: &SparkSession,
+) -> Option<Py<PySparkSession>> {
+    registry
+        .get(session.session_id())?
+        .iter()
+        .filter_map(|weak| live_session_object(py, weak))
+        .find(|object| {
+            object
+                .bind(py)
+                .try_borrow()
+                .is_ok_and(|object| object.session.is_same_session(session))
+        })
+}
+
+/// The Python object of `session`, one per session as PySpark has them: its 
live object, or
+/// else a new one, which the registry holds weakly. So the DataFrames, the 
streams and the
+/// clients of a session give back the session's object, as PySpark's do.
+pub(crate) fn session_object(
+    py: Python<'_>,
+    session: &SparkSession,
+) -> PyResult<Py<PySparkSession>> {
+    {
+        let registry = session_objects().lock().unwrap_or_else(|e| 
e.into_inner());
+        if let Some(object) = registered_session_object(py, &registry, 
session) {
+            return Ok(object);
+        }
+    }
+    let object = Py::new(py, PySparkSession::new(session.clone()))?;
+    let weak = py
+        .import("weakref")?
+        .call_method1("ref", (&object,))?
+        .unbind();
+    let mut registry = session_objects().lock().unwrap_or_else(|e| 
e.into_inner());
+    // Another thread may have made the object of this session meanwhile; then 
it is that one.
+    if let Some(existing) = registered_session_object(py, &registry, session) {
+        return Ok(existing);
+    }
+    registry.retain(|_, objects| {
+        objects.retain(|weak| live_session_object(py, weak).is_some());
+        !objects.is_empty()
+    });
+    registry
+        .entry(session.session_id().to_string())
+        .or_default()
+        .push(weak);
+    Ok(object)
+}
+
 /// Make `session` the default session, and the active session of this thread, 
when they are
 /// not set, as PySpark's `SparkSession._set_default_and_active_session`.
 fn set_default_and_active(py: Python<'_>, session: &Py<PySparkSession>) {
@@ -2253,8 +2325,11 @@ impl PySparkSession {
     ///
     /// Mirrors `SparkSession.dataSource.register`. Used to register custom 
data sources.
     #[getter]
-    fn dataSource(&self) -> PyDataSourceRegistration {
-        
PyDataSourceRegistration::new(PySparkSession::new(self.session.clone()))
+    fn dataSource(&self, py: Python<'_>) -> PyResult<PyDataSourceRegistration> 
{
+        Ok(PyDataSourceRegistration::new(session_object(
+            py,
+            &self.session,
+        )?))
     }
 
     /// Build and register a resource profile with the server.
@@ -2443,8 +2518,8 @@ impl PySparkSession {
     }
 
     #[pyo3(name = "newSession")]
-    fn new_session(&self) -> PySparkSession {
-        PySparkSession::new(self.session.new_session())
+    fn new_session(&self, py: Python<'_>) -> PyResult<Py<PySparkSession>> {
+        session_object(py, &self.session.new_session())
     }
 
     /// Clone this session on the server, as PySpark's 
`cloneSession(new_session_id=None)`: a
@@ -2456,7 +2531,7 @@ impl PySparkSession {
         &self,
         py: Python<'_>,
         new_session_id: Option<String>,
-    ) -> PyResult<PySparkSession> {
+    ) -> PyResult<Py<PySparkSession>> {
         let session = py
             .detach(|| {
                 self.session
@@ -2464,7 +2539,7 @@ impl PySparkSession {
                     .map_err(Box::new)
             })
             .to_pyerr()?;
-        Ok(PySparkSession::new(session))
+        session_object(py, &session)
     }
 }
 
diff --git a/crates/pyspark-rs/src/streaming.rs 
b/crates/pyspark-rs/src/streaming.rs
index 58a3c9c4..8376be3b 100644
--- a/crates/pyspark-rs/src/streaming.rs
+++ b/crates/pyspark-rs/src/streaming.rs
@@ -1245,7 +1245,7 @@ impl PyStreamingQueryManager {
     /// listener added twice is in the bus twice. Should the server fail to 
add its listener, this
     /// warns, as PySpark Connect's does.
     fn addListener(&self, py: Python<'_>, listener: Py<PyAny>) -> PyResult<()> 
{
-        let session = 
crate::session::PySparkSession::new(self.session.clone());
+        let session = crate::session::session_object(py, &self.session)?;
         listener
             .bind(py)
             .call_method1("_set_spark_session", (session,))?;
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 6dc8d152..de616a79 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -394,6 +394,12 @@ impl SparkSession {
         &self.client
     }
 
+    /// Whether `other` is this session or a clone of it, which shares its 
client. Two sessions
+    /// made apart are not the same session, even of the same session ID.
+    pub fn is_same_session(&self, other: &SparkSession) -> bool {
+        Arc::ptr_eq(&self.client, &other.client)
+    }
+
     /// Create a builder for a new SparkSession.
     pub fn builder() -> SparkSessionBuilder {
         SparkSessionBuilder::new()
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 252cd1ae..36900270 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1068,6 +1068,61 @@ def 
test_session_builders_set_their_options_and_validate_their_urls_as_pyspark(m
 
 
 
+def test_dataframes_give_back_their_session_object_as_pyspark(offline_spark):
+    # As PySpark: the DataFrames of a session, its streams and its client give 
back the session
+    # object itself, and two sessions do not mix. The registry holds the 
session objects
+    # weakly, so a session that is dropped is freed.
+    import gc
+    import weakref
+
+    def check(actual, expected):
+        # Not an assert: pytest reprs a DataFrame in a failed assert, which 
asks the server.
+        if actual is not expected:
+            raise AssertionError("not the session object that PySpark gives 
back")
+
+    spark = offline_spark
+    df = spark.range(2)
+    check(df.sparkSession, spark)
+    check(df._session, spark)
+    check(df.select("id").sparkSession, spark)
+    check(df.groupBy("id").count().sparkSession, spark)
+    check(df.na.drop().sparkSession, spark)
+    check(spark.read.format("rate").load().sparkSession, spark)
+    check(spark.udf.sparkSession, spark)
+    check(spark.udtf.sparkSession, spark)
+    check(spark.dataSource.sparkSession, spark)
+    assert df.sparkSession == spark
+    check(df.sparkSession.streams, spark.streams)
+    check(df.sparkSession.client, spark.client)
+    other = spark.newSession()
+    check(other.range(1).sparkSession, other)
+    if other.range(1).sparkSession is spark:
+        raise AssertionError("two sessions mixed")
+    check(other.range(1).sparkSession.streams, other.streams)
+    if other.streams is spark.streams:
+        raise AssertionError("two sessions share their streams")
+    ref = weakref.ref(other)
+    del other
+    gc.collect()
+    assert ref() is None
+    check(df.sparkSession, spark)
+    # Two sessions made apart, of the same session ID, are two objects, each 
that of its own
+    # DataFrames. create() sets the default session, so this runs in a process 
of its own.
+    import subprocess
+    import sys
+
+    subprocess.run([sys.executable, "-c", """
+import uuid
+from pyspark.sql import SparkSession
+url = f"sc://localhost:1/;session_id={uuid.uuid4()}"
+first = SparkSession.builder.remote(url).create()
+second = SparkSession.builder.remote(url).create()
+assert first is not second
+assert first.range(1).sparkSession is first
+assert second.range(1).sparkSession is second
+"""], check=True)
+
+
 def test_session_builder_class_constructor():
     from pyspark.sql import SparkSession
     b = SparkSession.Builder()


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to