This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new fb049ae  [SPARK-61129][CONNECT][PYTHON] Make SparkSession and 
DataFrame picklable, of the active session, in the Python client as PySpark does
fb049ae is described below

commit fb049aea926174a44e80bca490599fefaf655730
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 07:02:48 2026 +0900

    [SPARK-61129][CONNECT][PYTHON] Make SparkSession and DataFrame picklable, 
of the active session, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61129](https://issues.apache.org/jira/browse/SPARK-61129): The 
client's `SparkSession` and `DataFrame` now pickle with cloudpickle, as PySpark 
Connect's do:
    
    - **`SparkSession`** pickles as a function of its ID, which cloudpickle 
pickles by value. The function gives back the active session of that ID through 
the new `SparkSession._getActiveSessionIfMatches`:
      - Where no session is active, it raises `PySparkRuntimeError` 
`NO_ACTIVE_SESSION`, as on a Python worker of the server.
      - Where another session is active, it raises `PySparkAssertionError`.
    
      The function reads the class of `pyspark.sql.connect.session` when it is 
called, so the server's worker, which runs PySpark itself, raises the same 
error.
    - **`DataFrame`** pickles of its session and the bytes of its relation, 
PySpark's plan. The session unpickles first, so on the server's worker a 
pickled DataFrame raises `NO_ACTIVE_SESSION`, as PySpark's does.
    - **The core** rebuilds the DataFrame of a relation that its session sent 
with the plan ID of that relation (`DataFrame::from_sent_relation`). So the 
copy of a DataFrame is sent exactly as the DataFrame was, and the columns of 
the original resolve against it. PySpark's copies keep their plan IDs the same 
way.
    
    Plain `pickle` still cannot pickle either, as with PySpark: the function is 
a local one.
    
    ### Why are the changes needed?
    
    The client could pickle neither: `TypeError: cannot pickle 
'pyspark.sql.session.SparkSession' object`, and the same for its DataFrame. So 
a UDF or a UDTF that captured the session or a DataFrame failed on the client, 
while PySpark sends it and its worker raises `NO_ACTIVE_SESSION`. Apache 
Spark's `test_udtf_access_spark_session_connect` failed on this client.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes:
    - A session and a DataFrame can be pickled with cloudpickle, as with 
PySpark 4.3.0.
    - A UDF or a UDTF that captures either raises `PythonException` 
`NO_ACTIVE_SESSION` from the server, rather than a pickling error on the client.
    
    ### How was this patch tested?
    
    - A new offline test checks:
      - the round trips of a session, which is the same object, and of a 
DataFrame, whose relation is the same
      - the `NO_ACTIVE_SESSION` of either in a thread without an active session
      - the `PySparkAssertionError` of another session ID
      - that plain `pickle` refuses both
    - A new live test runs a UDTF that captures a DataFrame and a UDF that 
captures the session, which both raise `PythonException` `NO_ACTIVE_SESSION`. 
It passes with official PySpark 4.3.0 and fails without this change.
    - A new Rust test (`raw_relation.rs`) checks that the DataFrame of a 
relation that its session sent is sent as the relation was, and that the 
columns of the original refer to it.
    - Apache Spark's `test_udtf_access_spark_session_connect`, of 
`ArrowUDTFParityTests` and `LegacyArrowUDTFParityTests` in 
`test_parity_udtf.py`, now passes on this client against a local server.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2189 passed, 8 skipped), and the 
live drop-in and Arrow suites (178 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #325 from HyukjinKwon/SPARK-61129-pickle-session-dataframe.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/dataframe.rs         | 28 ++++++++++++++++++++
 crates/pyspark-rs/src/session.rs           | 41 +++++++++++++++++++++++++++++
 crates/spark-connect/src/dataframe.rs      | 17 ++++++++++++
 crates/spark-connect/src/plan.rs           | 10 +++++++
 crates/spark-connect/tests/raw_relation.rs | 23 ++++++++++++++++
 python/pyspark/sql/connect/dataframe.py    | 15 +++++++++++
 python/pyspark/sql/connect/session.py      | 16 ++++++++++++
 python/tests/test_dropin_e2e.py            | 27 +++++++++++++++++++
 python/tests/test_dropin_offline.py        | 42 ++++++++++++++++++++++++++++++
 9 files changed, 219 insertions(+)

diff --git a/crates/pyspark-rs/src/dataframe.rs 
b/crates/pyspark-rs/src/dataframe.rs
index 144d6fe..54033a3 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -578,6 +578,34 @@ impl PyDataFrame {
         pyo3::types::PyBytes::new(py, 
&self.dataframe.to_relation().encode_to_vec())
     }
 
+    /// A DataFrame pickles, as PySpark Connect's, of its session, which gives 
back the active
+    /// session of its ID, and of its plan, here the bytes of its relation. 
The session unpickles
+    /// first: so on a Python worker of the server, which has no active 
session, a pickled
+    /// DataFrame raises `NO_ACTIVE_SESSION`, as PySpark's does.
+    fn __reduce__<'py>(slf: &Bound<'py, Self>) -> PyResult<Bound<'py, PyAny>> {
+        let py = slf.py();
+        let this = slf.borrow();
+        py.import("pyspark.sql.connect.dataframe")?
+            .getattr("_reduce_dataframe")?
+            .call1((this.spark_session(), this._relation_bytes(py)))
+    }
+
+    /// The DataFrame of `relation`, the bytes of a relation of `session`, as 
a pickled DataFrame
+    /// unpickles.
+    #[staticmethod]
+    fn _of_relation(
+        session: &Bound<'_, crate::session::PySparkSession>,
+        relation: &[u8],
+    ) -> PyResult<PyDataFrame> {
+        use prost::Message;
+        let relation = spark_connect_proto::Relation::decode(relation)
+            .map_err(|error| 
pyo3::exceptions::PyValueError::new_err(error.to_string()))?;
+        Ok(PyDataFrame::new(DataFrame::from_sent_relation(
+            session.borrow().session.clone(),
+            relation,
+        )))
+    }
+
     /// The plan of this DataFrame, as PySpark Connect's `DataFrame._plan`, 
whose
     /// `plan(client)` is its relation, over which `pyspark.ml` builds its 
plans
     /// (`pyspark.sql.connect.plan`).
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 00e1a91..89b3d27 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -2068,6 +2068,47 @@ impl PySparkSession {
         active_session(py)
     }
 
+    /// The active session of this thread, if its ID is `session_id`, as 
PySpark Connect's
+    /// `SparkSession._getActiveSessionIfMatches`, which gives back a pickled 
session: without
+    /// an active session, as on a Python worker of the server, it raises 
`NO_ACTIVE_SESSION`,
+    /// and for another one, `PySparkAssertionError`.
+    #[staticmethod]
+    #[pyo3(name = "_getActiveSessionIfMatches")]
+    fn get_active_session_if_matches(
+        py: Python<'_>,
+        session_id: &str,
+    ) -> PyResult<Py<PySparkSession>> {
+        let Some(session) = active_session(py) else {
+            return Err(crate::column::raise_pyspark(
+                py,
+                "PySparkRuntimeError",
+                "NO_ACTIVE_SESSION",
+                &[],
+            ));
+        };
+        let active_id = session.borrow(py).session.session_id().to_string();
+        if active_id != session_id {
+            let error = py
+                .import("pyspark.errors")?
+                .getattr("PySparkAssertionError")?
+                .call1((format!(
+                    "Expected session ID does not match active session ID: \
+                     {session_id} != {active_id}"
+                ),))?;
+            return Err(PyErr::from_value(error));
+        }
+        Ok(session)
+    }
+
+    /// A session pickles, as PySpark Connect's: as a function of its ID, 
which cloudpickle
+    /// pickles by value, that gives back the active session of that ID
+    /// (`_getActiveSessionIfMatches`).
+    fn __reduce__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        py.import("pyspark.sql.connect.session")?
+            .getattr("_reduce_session")?
+            .call1((self.session.session_id().to_string(),))
+    }
+
     /// The type of the DDL string `ddl`, as the server parses it, as PySpark 
Connect's
     /// `SparkSession._parse_ddl`, of a `DDLParse` of an `AnalyzePlanRequest`.
     fn _parse_ddl(&self, py: Python<'_>, ddl: &str) -> PyResult<Py<PyAny>> {
diff --git a/crates/spark-connect/src/dataframe.rs 
b/crates/spark-connect/src/dataframe.rs
index b8dabbe..daac295 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1007,6 +1007,23 @@ impl DataFrame {
         )
     }
 
+    /// The DataFrame of `relation`, a relation that `session` sent, such as 
that of a DataFrame
+    /// that was pickled, which keeps its plan ID, as PySpark Connect's 
pickled plans keep
+    /// theirs: so it is sent as it is, and the columns of the DataFrame it 
was of resolve
+    /// against it. Not a stable API.
+    #[doc(hidden)]
+    pub fn from_sent_relation(session: SparkSession, relation: 
proto::Relation) -> DataFrame {
+        let plan_id = relation.common.as_ref().and_then(|common| 
common.plan_id);
+        let dataframe = DataFrame::from_relation(session, relation, None);
+        if let Some(plan_id) = plan_id {
+            dataframe
+                .session
+                .plan_ids()
+                .assign(&dataframe.plan, plan_id);
+        }
+        dataframe
+    }
+
     /// The relation of this DataFrame as it is sent, with the plan IDs of its 
session, for a
     /// plan built outside the core over it, such as of `pyspark.ml`. Not a 
stable API.
     #[doc(hidden)]
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 4d03854..559c81b 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -547,6 +547,16 @@ impl PlanIds {
         id
     }
 
+    /// Give `node` the plan ID `id`, of a relation that this session sent: so 
a DataFrame of the
+    /// relation, such as one that was pickled, keeps the plan ID of the 
DataFrame it was of.
+    pub(crate) fn assign(&self, node: &Arc<LogicalPlan>, id: i64) {
+        let mut nodes = self.nodes.lock().unwrap_or_else(|e| e.into_inner());
+        nodes.next = nodes.next.max(id.saturating_add(1));
+        nodes
+            .ids
+            .insert(Arc::as_ptr(node) as usize, (Arc::downgrade(node), id));
+    }
+
     /// The live node of the plan ID `id`, as the server reports a relation by 
its plan ID.
     pub(crate) fn node_of(&self, id: i64) -> Option<Arc<LogicalPlan>> {
         let nodes = self.nodes.lock().unwrap_or_else(|e| e.into_inner());
diff --git a/crates/spark-connect/tests/raw_relation.rs 
b/crates/spark-connect/tests/raw_relation.rs
index 037a9dd..90dfe2d 100644
--- a/crates/spark-connect/tests/raw_relation.rs
+++ b/crates/spark-connect/tests/raw_relation.rs
@@ -96,6 +96,29 @@ fn 
raw_relations_keep_their_owners_alive_while_a_dataframe_uses_them() {
     assert!(alive.upgrade().is_none());
 }
 
+#[test]
+fn sent_relations_keep_the_plan_ids_of_the_dataframes_they_were_of() {
+    // As a pickled DataFrame of PySpark Connect, whose plan keeps its plan 
IDs: the DataFrame of
+    // a relation that its session sent is sent as it is, and the columns of 
the DataFrame it
+    // was of refer to it.
+    let session = offline();
+    let df = session.range(3).unwrap().select([col("id").alias("x")]);
+    let relation = df.to_relation();
+    let plan_id = relation.common.as_ref().and_then(|common| common.plan_id);
+    let copy = DataFrame::from_sent_relation(session.clone(), 
relation.clone());
+    assert_eq!(copy.to_relation(), relation);
+    assert_eq!(copy.clone().to_relation(), relation);
+    let filtered = copy.filter(df.col("x").gt(lit(0)));
+    let Some(proto::relation::RelType::Filter(filter)) = 
filtered.to_relation().rel_type else {
+        panic!("expected a filter");
+    };
+    assert_eq!(filter.input.as_deref(), Some(&relation));
+    assert!(format!("{:?}", filter.condition).contains(&format!("plan_id: 
{:?}", plan_id)));
+    // The other plans of the session take other plan IDs.
+    let other = session.range(1).unwrap().to_relation();
+    assert_ne!(other.common.and_then(|common| common.plan_id), plan_id);
+}
+
 #[test]
 fn commands_of_no_command_type_are_errors() {
     let error = offline()
diff --git a/python/pyspark/sql/connect/dataframe.py 
b/python/pyspark/sql/connect/dataframe.py
index 693c5e9..5297f0d 100644
--- a/python/pyspark/sql/connect/dataframe.py
+++ b/python/pyspark/sql/connect/dataframe.py
@@ -15,7 +15,22 @@
 # limitations under the License.
 #
 """Rust-backed DataFrame under the connect import path (testing/connectutils 
imports from here)."""
+from typing import Any, Callable, Tuple
+
 from pyspark._pyspark import DataFrame
 from pyspark.sql.dataframe import DataFrameNaFunctions, DataFrameStatFunctions
 
 __all__ = ["DataFrame", "DataFrameNaFunctions", "DataFrameStatFunctions"]
+
+
+def _reduce_dataframe(session: Any, relation: bytes) -> Tuple[Callable[..., 
Any], Tuple[Any, ...]]:
+    """The ``__reduce__`` of a DataFrame of ``session`` and the bytes of its 
``relation``, as
+    PySpark Connect's, of its session and its plan: a function, which 
cloudpickle pickles by
+    value, of the session, which unpickles first, and of the relation."""
+
+    def creator(session: Any, relation: bytes) -> "DataFrame":
+        from pyspark import _pyspark
+
+        return _pyspark.DataFrame._of_relation(session, relation)
+
+    return creator, (session, relation)
diff --git a/python/pyspark/sql/connect/session.py 
b/python/pyspark/sql/connect/session.py
index 3f29279..0403bf1 100644
--- a/python/pyspark/sql/connect/session.py
+++ b/python/pyspark/sql/connect/session.py
@@ -15,6 +15,22 @@
 # limitations under the License.
 #
 """Rust-backed SparkSession under the connect import path 
(testing/connectutils imports from here)."""
+from typing import Callable, Tuple
+
 from pyspark.sql.session import SparkSession
 
 __all__ = ["SparkSession"]
+
+
+def _reduce_session(session_id: str) -> Tuple[Callable[[str], "SparkSession"], 
Tuple[str]]:
+    """The ``__reduce__`` of the session of ``session_id``, as PySpark 
Connect's: a function,
+    which cloudpickle pickles by value, that gives back the active session of 
that ID. It reads
+    the class of ``pyspark.sql.connect.session`` when it is called, so that on 
a Python worker
+    of the server, whose PySpark's has no active session, it raises 
``NO_ACTIVE_SESSION``."""
+
+    def creator(old_session_id: str) -> "SparkSession":
+        from pyspark.sql.connect.session import SparkSession
+
+        return SparkSession._getActiveSessionIfMatches(old_session_id)
+
+    return creator, (session_id,)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 19896c5..9e0391c 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -229,6 +229,33 @@ def 
test_repr_shows_the_rows_with_eager_evaluation_as_pyspark(session):
     assert repr(df) == "DataFrame[id: bigint]"
 
 
+def 
test_udtfs_and_udfs_of_a_dataframe_or_the_session_raise_no_active_session_as_pyspark(session):
+    # As PySpark Connect's: a UDTF or a UDF that captures a DataFrame or the 
session pickles it
+    # of its session, which raises NO_ACTIVE_SESSION on the Python worker of 
the server, where
+    # no session is active.
+    from pyspark.errors import PythonException
+
+    df = session.range(10)
+
+    @F.udtf(returnType="x: int")
+    class Collect:
+        def eval(self):
+            df.collect()
+            yield (1,)
+
+    with pytest.raises(PythonException, match="NO_ACTIVE_SESSION"):
+        Collect().collect()
+
+    @F.udf("int")
+    def of_session(x):
+        return len(session.version) + x
+
+    with pytest.raises(PythonException, match="NO_ACTIVE_SESSION"):
+        session.range(1).select(of_session("id")).collect()
+
+
+
+
 def test_join_with_an_unsupported_join_type_raises_as_pyspark(session):
     # As PySpark: the join type is checked when the join is built, which 
raises the
     # AnalysisException of UNSUPPORTED_JOIN_TYPE, naming the type as it is 
normalized.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 3bc8f45..397b587 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -5128,6 +5128,48 @@ def test_spark_conf_chain_operations():
     assert len(conf.getAll()) == 4
 
 
+def 
test_sessions_and_dataframes_pickle_of_the_active_session_as_pyspark(mock_connect):
+    # As PySpark Connect's, with cloudpickle: a session pickles as a function 
of its ID, which
+    # gives back the active session of that ID, and a DataFrame of its session 
and its plan, of
+    # its plan IDs, so that it is sent as it is. Where no session is active, 
as on a Python
+    # worker of the server, both raise NO_ACTIVE_SESSION, as the session 
unpickles first; where
+    # another is, PySparkAssertionError. Plain pickle cannot pickle the 
function, a local one.
+    import pickle
+    import threading
+
+    from pyspark.errors import PySparkAssertionError, PySparkRuntimeError
+    from pyspark.serializers import CloudPickleSerializer
+
+    spark, _ = mock_connect
+    serializer = CloudPickleSerializer()
+    assert serializer.loads(serializer.dumps(spark)) is spark
+    df = spark.range(3).selectExpr("id * 2 AS x")
+    copy = serializer.loads(serializer.dumps(df))
+    assert type(copy) is type(df) and copy._relation_bytes() == 
df._relation_bytes()
+    assert copy.sparkSession.session_id == spark.session_id
+    for obj in (spark, df):
+        with pytest.raises((AttributeError, pickle.PicklingError)):
+            pickle.dumps(obj)
+
+    errors = []
+
+    def unpickle(data):
+        try:
+            serializer.loads(data)
+        except PySparkRuntimeError as error:
+            errors.append(error.getCondition())
+
+    for obj in (spark, df):
+        thread = threading.Thread(target=unpickle, 
args=(serializer.dumps(obj),))
+        thread.start()
+        thread.join()
+    assert errors == ["NO_ACTIVE_SESSION", "NO_ACTIVE_SESSION"]
+    with pytest.raises(PySparkAssertionError, match="Expected session ID does 
not match"):
+        SparkSession._getActiveSessionIfMatches("another")
+
+
+
+
 def 
test_inheritable_threads_inherit_the_tags_of_the_session_as_pyspark(mock_connect):
     # As PySpark's of Spark Connect: InheritableThread takes the tags of its 
session of the
     # thread that starts it, and inheritable_thread_target those of the thread 
that wraps the


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to