This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new fb049ae [SPARK-61129][CONNECT][PYTHON] Make SparkSession and
DataFrame picklable, of the active session, in the Python client as PySpark does
fb049ae is described below
commit fb049aea926174a44e80bca490599fefaf655730
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 07:02:48 2026 +0900
[SPARK-61129][CONNECT][PYTHON] Make SparkSession and DataFrame picklable,
of the active session, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61129](https://issues.apache.org/jira/browse/SPARK-61129): The
client's `SparkSession` and `DataFrame` now pickle with cloudpickle, as PySpark
Connect's do:
- **`SparkSession`** pickles as a function of its ID, which cloudpickle
pickles by value. The function gives back the active session of that ID through
the new `SparkSession._getActiveSessionIfMatches`:
- Where no session is active, it raises `PySparkRuntimeError`
`NO_ACTIVE_SESSION`, as on a Python worker of the server.
- Where another session is active, it raises `PySparkAssertionError`.
The function reads the class of `pyspark.sql.connect.session` when it is
called, so the server's worker, which runs PySpark itself, raises the same
error.
- **`DataFrame`** pickles of its session and the bytes of its relation,
PySpark's plan. The session unpickles first, so on the server's worker a
pickled DataFrame raises `NO_ACTIVE_SESSION`, as PySpark's does.
- **The core** rebuilds the DataFrame of a relation that its session sent
with the plan ID of that relation (`DataFrame::from_sent_relation`). So the
copy of a DataFrame is sent exactly as the DataFrame was, and the columns of
the original resolve against it. PySpark's copies keep their plan IDs the same
way.
Plain `pickle` still cannot pickle either, as with PySpark: the function is
a local one.
### Why are the changes needed?
The client could pickle neither: `TypeError: cannot pickle
'pyspark.sql.session.SparkSession' object`, and the same for its DataFrame. So
a UDF or a UDTF that captured the session or a DataFrame failed on the client,
while PySpark sends it and its worker raises `NO_ACTIVE_SESSION`. Apache
Spark's `test_udtf_access_spark_session_connect` failed on this client.
### Does this PR introduce _any_ user-facing change?
Yes:
- A session and a DataFrame can be pickled with cloudpickle, as with
PySpark 4.3.0.
- A UDF or a UDTF that captures either raises `PythonException`
`NO_ACTIVE_SESSION` from the server, rather than a pickling error on the client.
### How was this patch tested?
- A new offline test checks:
- the round trips of a session, which is the same object, and of a
DataFrame, whose relation is the same
- the `NO_ACTIVE_SESSION` of either in a thread without an active session
- the `PySparkAssertionError` of another session ID
- that plain `pickle` refuses both
- A new live test runs a UDTF that captures a DataFrame and a UDF that
captures the session, which both raise `PythonException` `NO_ACTIVE_SESSION`.
It passes with official PySpark 4.3.0 and fails without this change.
- A new Rust test (`raw_relation.rs`) checks that the DataFrame of a
relation that its session sent is sent as the relation was, and that the
columns of the original refer to it.
- Apache Spark's `test_udtf_access_spark_session_connect`, of
`ArrowUDTFParityTests` and `LegacyArrowUDTFParityTests` in
`test_parity_udtf.py`, now passes on this client against a local server.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2189 passed, 8 skipped), and the
live drop-in and Arrow suites (178 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #325 from HyukjinKwon/SPARK-61129-pickle-session-dataframe.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/dataframe.rs | 28 ++++++++++++++++++++
crates/pyspark-rs/src/session.rs | 41 +++++++++++++++++++++++++++++
crates/spark-connect/src/dataframe.rs | 17 ++++++++++++
crates/spark-connect/src/plan.rs | 10 +++++++
crates/spark-connect/tests/raw_relation.rs | 23 ++++++++++++++++
python/pyspark/sql/connect/dataframe.py | 15 +++++++++++
python/pyspark/sql/connect/session.py | 16 ++++++++++++
python/tests/test_dropin_e2e.py | 27 +++++++++++++++++++
python/tests/test_dropin_offline.py | 42 ++++++++++++++++++++++++++++++
9 files changed, 219 insertions(+)
diff --git a/crates/pyspark-rs/src/dataframe.rs
b/crates/pyspark-rs/src/dataframe.rs
index 144d6fe..54033a3 100644
--- a/crates/pyspark-rs/src/dataframe.rs
+++ b/crates/pyspark-rs/src/dataframe.rs
@@ -578,6 +578,34 @@ impl PyDataFrame {
pyo3::types::PyBytes::new(py,
&self.dataframe.to_relation().encode_to_vec())
}
+ /// A DataFrame pickles, as PySpark Connect's, of its session, which gives
back the active
+ /// session of its ID, and of its plan, here the bytes of its relation.
The session unpickles
+ /// first: so on a Python worker of the server, which has no active
session, a pickled
+ /// DataFrame raises `NO_ACTIVE_SESSION`, as PySpark's does.
+ fn __reduce__<'py>(slf: &Bound<'py, Self>) -> PyResult<Bound<'py, PyAny>> {
+ let py = slf.py();
+ let this = slf.borrow();
+ py.import("pyspark.sql.connect.dataframe")?
+ .getattr("_reduce_dataframe")?
+ .call1((this.spark_session(), this._relation_bytes(py)))
+ }
+
+ /// The DataFrame of `relation`, the bytes of a relation of `session`, as
a pickled DataFrame
+ /// unpickles.
+ #[staticmethod]
+ fn _of_relation(
+ session: &Bound<'_, crate::session::PySparkSession>,
+ relation: &[u8],
+ ) -> PyResult<PyDataFrame> {
+ use prost::Message;
+ let relation = spark_connect_proto::Relation::decode(relation)
+ .map_err(|error|
pyo3::exceptions::PyValueError::new_err(error.to_string()))?;
+ Ok(PyDataFrame::new(DataFrame::from_sent_relation(
+ session.borrow().session.clone(),
+ relation,
+ )))
+ }
+
/// The plan of this DataFrame, as PySpark Connect's `DataFrame._plan`,
whose
/// `plan(client)` is its relation, over which `pyspark.ml` builds its
plans
/// (`pyspark.sql.connect.plan`).
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 00e1a91..89b3d27 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -2068,6 +2068,47 @@ impl PySparkSession {
active_session(py)
}
+ /// The active session of this thread, if its ID is `session_id`, as
PySpark Connect's
+ /// `SparkSession._getActiveSessionIfMatches`, which gives back a pickled
session: without
+ /// an active session, as on a Python worker of the server, it raises
`NO_ACTIVE_SESSION`,
+ /// and for another one, `PySparkAssertionError`.
+ #[staticmethod]
+ #[pyo3(name = "_getActiveSessionIfMatches")]
+ fn get_active_session_if_matches(
+ py: Python<'_>,
+ session_id: &str,
+ ) -> PyResult<Py<PySparkSession>> {
+ let Some(session) = active_session(py) else {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkRuntimeError",
+ "NO_ACTIVE_SESSION",
+ &[],
+ ));
+ };
+ let active_id = session.borrow(py).session.session_id().to_string();
+ if active_id != session_id {
+ let error = py
+ .import("pyspark.errors")?
+ .getattr("PySparkAssertionError")?
+ .call1((format!(
+ "Expected session ID does not match active session ID: \
+ {session_id} != {active_id}"
+ ),))?;
+ return Err(PyErr::from_value(error));
+ }
+ Ok(session)
+ }
+
+ /// A session pickles, as PySpark Connect's: as a function of its ID,
which cloudpickle
+ /// pickles by value, that gives back the active session of that ID
+ /// (`_getActiveSessionIfMatches`).
+ fn __reduce__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+ py.import("pyspark.sql.connect.session")?
+ .getattr("_reduce_session")?
+ .call1((self.session.session_id().to_string(),))
+ }
+
/// The type of the DDL string `ddl`, as the server parses it, as PySpark
Connect's
/// `SparkSession._parse_ddl`, of a `DDLParse` of an `AnalyzePlanRequest`.
fn _parse_ddl(&self, py: Python<'_>, ddl: &str) -> PyResult<Py<PyAny>> {
diff --git a/crates/spark-connect/src/dataframe.rs
b/crates/spark-connect/src/dataframe.rs
index b8dabbe..daac295 100644
--- a/crates/spark-connect/src/dataframe.rs
+++ b/crates/spark-connect/src/dataframe.rs
@@ -1007,6 +1007,23 @@ impl DataFrame {
)
}
+ /// The DataFrame of `relation`, a relation that `session` sent, such as
that of a DataFrame
+ /// that was pickled, which keeps its plan ID, as PySpark Connect's
pickled plans keep
+ /// theirs: so it is sent as it is, and the columns of the DataFrame it
was of resolve
+ /// against it. Not a stable API.
+ #[doc(hidden)]
+ pub fn from_sent_relation(session: SparkSession, relation:
proto::Relation) -> DataFrame {
+ let plan_id = relation.common.as_ref().and_then(|common|
common.plan_id);
+ let dataframe = DataFrame::from_relation(session, relation, None);
+ if let Some(plan_id) = plan_id {
+ dataframe
+ .session
+ .plan_ids()
+ .assign(&dataframe.plan, plan_id);
+ }
+ dataframe
+ }
+
/// The relation of this DataFrame as it is sent, with the plan IDs of its
session, for a
/// plan built outside the core over it, such as of `pyspark.ml`. Not a
stable API.
#[doc(hidden)]
diff --git a/crates/spark-connect/src/plan.rs b/crates/spark-connect/src/plan.rs
index 4d03854..559c81b 100644
--- a/crates/spark-connect/src/plan.rs
+++ b/crates/spark-connect/src/plan.rs
@@ -547,6 +547,16 @@ impl PlanIds {
id
}
+ /// Give `node` the plan ID `id`, of a relation that this session sent: so
a DataFrame of the
+ /// relation, such as one that was pickled, keeps the plan ID of the
DataFrame it was of.
+ pub(crate) fn assign(&self, node: &Arc<LogicalPlan>, id: i64) {
+ let mut nodes = self.nodes.lock().unwrap_or_else(|e| e.into_inner());
+ nodes.next = nodes.next.max(id.saturating_add(1));
+ nodes
+ .ids
+ .insert(Arc::as_ptr(node) as usize, (Arc::downgrade(node), id));
+ }
+
/// The live node of the plan ID `id`, as the server reports a relation by
its plan ID.
pub(crate) fn node_of(&self, id: i64) -> Option<Arc<LogicalPlan>> {
let nodes = self.nodes.lock().unwrap_or_else(|e| e.into_inner());
diff --git a/crates/spark-connect/tests/raw_relation.rs
b/crates/spark-connect/tests/raw_relation.rs
index 037a9dd..90dfe2d 100644
--- a/crates/spark-connect/tests/raw_relation.rs
+++ b/crates/spark-connect/tests/raw_relation.rs
@@ -96,6 +96,29 @@ fn
raw_relations_keep_their_owners_alive_while_a_dataframe_uses_them() {
assert!(alive.upgrade().is_none());
}
+#[test]
+fn sent_relations_keep_the_plan_ids_of_the_dataframes_they_were_of() {
+ // As a pickled DataFrame of PySpark Connect, whose plan keeps its plan
IDs: the DataFrame of
+ // a relation that its session sent is sent as it is, and the columns of
the DataFrame it
+ // was of refer to it.
+ let session = offline();
+ let df = session.range(3).unwrap().select([col("id").alias("x")]);
+ let relation = df.to_relation();
+ let plan_id = relation.common.as_ref().and_then(|common| common.plan_id);
+ let copy = DataFrame::from_sent_relation(session.clone(),
relation.clone());
+ assert_eq!(copy.to_relation(), relation);
+ assert_eq!(copy.clone().to_relation(), relation);
+ let filtered = copy.filter(df.col("x").gt(lit(0)));
+ let Some(proto::relation::RelType::Filter(filter)) =
filtered.to_relation().rel_type else {
+ panic!("expected a filter");
+ };
+ assert_eq!(filter.input.as_deref(), Some(&relation));
+ assert!(format!("{:?}", filter.condition).contains(&format!("plan_id:
{:?}", plan_id)));
+ // The other plans of the session take other plan IDs.
+ let other = session.range(1).unwrap().to_relation();
+ assert_ne!(other.common.and_then(|common| common.plan_id), plan_id);
+}
+
#[test]
fn commands_of_no_command_type_are_errors() {
let error = offline()
diff --git a/python/pyspark/sql/connect/dataframe.py
b/python/pyspark/sql/connect/dataframe.py
index 693c5e9..5297f0d 100644
--- a/python/pyspark/sql/connect/dataframe.py
+++ b/python/pyspark/sql/connect/dataframe.py
@@ -15,7 +15,22 @@
# limitations under the License.
#
"""Rust-backed DataFrame under the connect import path (testing/connectutils
imports from here)."""
+from typing import Any, Callable, Tuple
+
from pyspark._pyspark import DataFrame
from pyspark.sql.dataframe import DataFrameNaFunctions, DataFrameStatFunctions
__all__ = ["DataFrame", "DataFrameNaFunctions", "DataFrameStatFunctions"]
+
+
+def _reduce_dataframe(session: Any, relation: bytes) -> Tuple[Callable[...,
Any], Tuple[Any, ...]]:
+ """The ``__reduce__`` of a DataFrame of ``session`` and the bytes of its
``relation``, as
+ PySpark Connect's, of its session and its plan: a function, which
cloudpickle pickles by
+ value, of the session, which unpickles first, and of the relation."""
+
+ def creator(session: Any, relation: bytes) -> "DataFrame":
+ from pyspark import _pyspark
+
+ return _pyspark.DataFrame._of_relation(session, relation)
+
+ return creator, (session, relation)
diff --git a/python/pyspark/sql/connect/session.py
b/python/pyspark/sql/connect/session.py
index 3f29279..0403bf1 100644
--- a/python/pyspark/sql/connect/session.py
+++ b/python/pyspark/sql/connect/session.py
@@ -15,6 +15,22 @@
# limitations under the License.
#
"""Rust-backed SparkSession under the connect import path
(testing/connectutils imports from here)."""
+from typing import Callable, Tuple
+
from pyspark.sql.session import SparkSession
__all__ = ["SparkSession"]
+
+
+def _reduce_session(session_id: str) -> Tuple[Callable[[str], "SparkSession"],
Tuple[str]]:
+ """The ``__reduce__`` of the session of ``session_id``, as PySpark
Connect's: a function,
+ which cloudpickle pickles by value, that gives back the active session of
that ID. It reads
+ the class of ``pyspark.sql.connect.session`` when it is called, so that on
a Python worker
+ of the server, whose PySpark's has no active session, it raises
``NO_ACTIVE_SESSION``."""
+
+ def creator(old_session_id: str) -> "SparkSession":
+ from pyspark.sql.connect.session import SparkSession
+
+ return SparkSession._getActiveSessionIfMatches(old_session_id)
+
+ return creator, (session_id,)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 19896c5..9e0391c 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -229,6 +229,33 @@ def
test_repr_shows_the_rows_with_eager_evaluation_as_pyspark(session):
assert repr(df) == "DataFrame[id: bigint]"
+def
test_udtfs_and_udfs_of_a_dataframe_or_the_session_raise_no_active_session_as_pyspark(session):
+ # As PySpark Connect's: a UDTF or a UDF that captures a DataFrame or the
session pickles it
+ # of its session, which raises NO_ACTIVE_SESSION on the Python worker of
the server, where
+ # no session is active.
+ from pyspark.errors import PythonException
+
+ df = session.range(10)
+
+ @F.udtf(returnType="x: int")
+ class Collect:
+ def eval(self):
+ df.collect()
+ yield (1,)
+
+ with pytest.raises(PythonException, match="NO_ACTIVE_SESSION"):
+ Collect().collect()
+
+ @F.udf("int")
+ def of_session(x):
+ return len(session.version) + x
+
+ with pytest.raises(PythonException, match="NO_ACTIVE_SESSION"):
+ session.range(1).select(of_session("id")).collect()
+
+
+
+
def test_join_with_an_unsupported_join_type_raises_as_pyspark(session):
# As PySpark: the join type is checked when the join is built, which
raises the
# AnalysisException of UNSUPPORTED_JOIN_TYPE, naming the type as it is
normalized.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 3bc8f45..397b587 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -5128,6 +5128,48 @@ def test_spark_conf_chain_operations():
assert len(conf.getAll()) == 4
+def
test_sessions_and_dataframes_pickle_of_the_active_session_as_pyspark(mock_connect):
+ # As PySpark Connect's, with cloudpickle: a session pickles as a function
of its ID, which
+ # gives back the active session of that ID, and a DataFrame of its session
and its plan, of
+ # its plan IDs, so that it is sent as it is. Where no session is active,
as on a Python
+ # worker of the server, both raise NO_ACTIVE_SESSION, as the session
unpickles first; where
+ # another is, PySparkAssertionError. Plain pickle cannot pickle the
function, a local one.
+ import pickle
+ import threading
+
+ from pyspark.errors import PySparkAssertionError, PySparkRuntimeError
+ from pyspark.serializers import CloudPickleSerializer
+
+ spark, _ = mock_connect
+ serializer = CloudPickleSerializer()
+ assert serializer.loads(serializer.dumps(spark)) is spark
+ df = spark.range(3).selectExpr("id * 2 AS x")
+ copy = serializer.loads(serializer.dumps(df))
+ assert type(copy) is type(df) and copy._relation_bytes() ==
df._relation_bytes()
+ assert copy.sparkSession.session_id == spark.session_id
+ for obj in (spark, df):
+ with pytest.raises((AttributeError, pickle.PicklingError)):
+ pickle.dumps(obj)
+
+ errors = []
+
+ def unpickle(data):
+ try:
+ serializer.loads(data)
+ except PySparkRuntimeError as error:
+ errors.append(error.getCondition())
+
+ for obj in (spark, df):
+ thread = threading.Thread(target=unpickle,
args=(serializer.dumps(obj),))
+ thread.start()
+ thread.join()
+ assert errors == ["NO_ACTIVE_SESSION", "NO_ACTIVE_SESSION"]
+ with pytest.raises(PySparkAssertionError, match="Expected session ID does
not match"):
+ SparkSession._getActiveSessionIfMatches("another")
+
+
+
+
def
test_inheritable_threads_inherit_the_tags_of_the_session_as_pyspark(mock_connect):
# As PySpark's of Spark Connect: InheritableThread takes the tags of its
session of the
# thread that starts it, and inheritable_thread_target those of the thread
that wraps the
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]