This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new caa0ecb2 [SPARK-61167][CONNECT][PYTHON] Pickle Columns as their 
expressions, also for the Python workers of the server, in the Python client as 
PySpark does
caa0ecb2 is described below

commit caa0ecb25749214b7f5800c58649c0a1ed428788
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 16:00:17 2026 +0900

    [SPARK-61167][CONNECT][PYTHON] Pickle Columns as their expressions, also 
for the Python workers of the server, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61167](https://issues.apache.org/jira/browse/SPARK-61167): PySpark 
Connect's Columns are Python objects of their expressions, which pickle and 
deepcopy, also with cloudpickle. So a `foreachBatch` function, which runs on a 
Python worker of the server, can capture a Column defined outside of it, and 
the worker unpickles it as PySpark Connect's.
    
    The client's Column now pickles as the bytes of its 
`spark.connect.Expression` and its rendering, of 
`pyspark.sql.connect._column_reduce.column_of`:
    - **On the client** it gives back a Column of the new 
`ExpressionNode::Sent` of the Rust core. That node sends the expression as it 
was sent, and renders as the Column did. The expressions over it send and 
render as those over the original.
    - **On a worker without the Rust client** it gives back PySpark Connect's 
Column, of an `Expression` of that proto. Everything that it calls there is of 
PySpark.
    - `pyspark.sql.connect.column` registers the module of the function with 
cloudpickle to pickle by value. So cloudpickle sends the function with a 
Column, while `pickle` pickles it by reference on the client, as PySpark 
Connect's Columns pickle with `pickle` too.
    
    A Column so unpickled is of an expression as it was sent. The methods that 
build on the kind of an expression take it as an expression of its own, on the 
client as on PySpark's worker, which do not know the kind of a sent expression 
either. Such methods are chaining `when` on an unpickled `when`, or sorting by 
an unpickled `desc()`, rather than calling `desc()` again.
    
    ### Why are the changes needed?
    
    The client's Column, of the Rust extension, could not be pickled:
    
    ```
    PySparkPicklingError: [STREAMING_CONNECT_SERIALIZATION_ERROR] Cannot 
serialize the function `foreachBatch`. ...
    ```
    
    `pickle.dumps(column)` and `copy.deepcopy(column)` raised `TypeError`. 
Apache Spark's `test_streaming_foreach_batch_external_column` of 
`test_parity_foreach_batch.py` failed on this client.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Columns pickle and deepcopy, and `foreachBatch` functions that capture 
them run, as with PySpark 4.3.0. For the Rust API, `ExpressionNode::Sent` and 
`SentExpression` are new.
    
    ### How was this patch tested?
    
    - A new Rust unit test checks that a sent expression sends and renders as 
it was, also under a function.
    - A new offline test pickles, cloudpickles and deep-copies six Columns, of 
a reference, an alias, a literal, `when`, a field and `desc()`. It checks their 
reprs and plans, and those of an expression over them. It passes with official 
PySpark 4.3.0, and fails without this change.
    - A new live test runs a `foreachBatch` function that captures a Column and 
writes a table of it, as Apache Spark's test does. It passes with official 
PySpark 4.3.0, and fails without this change.
    - Apache Spark's `test_parity_foreach_batch.py` passes on this client 
against a local server, all 23 tests.
    - The lib tests of both crates (598 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2220 passed, 8 skipped), and the 
live drop-in and Arrow suites (204 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #356 from HyukjinKwon/SPARK-61167-column-pickle.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/column.rs              | 31 +++++++++++++++
 crates/spark-connect/src/expression.rs       | 34 ++++++++++++++++
 python/pyspark/sql/connect/_column_reduce.py | 59 ++++++++++++++++++++++++++++
 python/pyspark/sql/connect/column.py         | 16 ++++++++
 python/tests/test_dropin_e2e.py              | 21 ++++++++++
 python/tests/test_dropin_offline.py          | 29 ++++++++++++++
 6 files changed, 190 insertions(+)

diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index 01fb5dfe..2225c1c6 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -679,6 +679,37 @@ impl PyColumn {
         self._expr().to_plan(py, session)
     }
 
+    /// A Column pickles, as PySpark Connect's, which pickles its expression: 
of the bytes of the
+    /// `spark.connect.Expression` of this column and of its rendering, of 
which the function of
+    /// `pyspark.sql.connect.column._reduce_column` gives back the Column, 
also on a Python worker
+    /// of the server, such as that of a `foreachBatch` function, as PySpark 
Connect's Column.
+    fn __reduce__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        use prost::Message;
+        let expression = pyo3::types::PyBytes::new(py, 
&self.column.to_proto().encode_to_vec());
+        py.import("pyspark.sql.connect.column")?
+            .getattr("_reduce_column")?
+            .call1((expression, self.column.expression().render()))
+    }
+
+    /// The Column of `expression`, the bytes of a `spark.connect.Expression`, 
and of its
+    /// `rendered` rendering, as a pickled Column unpickles: it sends the 
expression as it is.
+    #[staticmethod]
+    fn _of_expression(expression: &[u8], rendered: String) -> 
PyResult<PyColumn> {
+        use prost::Message;
+        let expression = spark_connect_proto::Expression::decode(expression)
+            .map_err(|error| 
pyo3::exceptions::PyValueError::new_err(error.to_string()))?;
+        Ok(PyColumn::new(Column::new(
+            spark_connect::expression::Expression::new(
+                spark_connect::expression::ExpressionNode::Sent(Box::new(
+                    spark_connect::expression::SentExpression {
+                        expression,
+                        rendered,
+                    },
+                )),
+            ),
+        )))
+    }
+
     fn __str__(&self) -> String {
         let _origin = crate::origin::with_origin("__str__");
         self.__repr__()
diff --git a/crates/spark-connect/src/expression.rs 
b/crates/spark-connect/src/expression.rs
index 6016b52b..bfa58b83 100644
--- a/crates/spark-connect/src/expression.rs
+++ b/crates/spark-connect/src/expression.rs
@@ -103,6 +103,17 @@ pub enum ExpressionNode {
     /// `pyspark.sql.connect.expressions.SubqueryExpression` - a scalar, 
EXISTS or IN
     /// subquery of a DataFrame, or a table argument of a table-valued 
function.
     Subquery(Box<SubqueryExpression>),
+    /// An expression as it was sent, of its `spark.connect.Expression` and 
its rendering, as a
+    /// pickled Column unpickles: the client sends the expression as it is.
+    Sent(Box<SentExpression>),
+}
+
+/// An expression of [`ExpressionNode::Sent`]: the `spark.connect.Expression` 
of a Column as it
+/// was sent, and its rendering, as `Column<'...'>` shows it.
+#[derive(Debug, Clone, PartialEq)]
+pub struct SentExpression {
+    pub expression: proto::Expression,
+    pub rendered: String,
 }
 
 /// The type of a [`SubqueryExpression`]. Mirrors the `subquery_type` of
@@ -404,6 +415,7 @@ impl Expression {
             // As `NamedArgumentExpression.__repr__`.
             ExpressionNode::NamedArgument(arg) => format!("{} => {}", arg.key, 
arg.value.render()),
             ExpressionNode::Subquery(subquery) => subquery.render(),
+            ExpressionNode::Sent(sent) => sent.rendered.clone(),
         }
     }
 
@@ -492,6 +504,7 @@ impl Expression {
                 ..Default::default()
             },
             ExpressionNode::Subquery(subquery) => subquery.to_proto(),
+            ExpressionNode::Sent(sent) => sent.expression.clone(),
         }
     }
 }
@@ -1656,6 +1669,27 @@ mod tests {
         }
     }
 
+    #[test]
+    fn sent_expressions_are_sent_and_rendered_as_they_were() {
+        // As a pickled Column unpickles: the expression as it was sent, of 
its rendering, which
+        // the expressions over it send and render.
+        let sum = Expression::function(
+            "+",
+            vec![col("a"), Expression::literal(LiteralExpression::Long(1))],
+        );
+        let original = 
Expression::new(ExpressionNode::Alias(Box::new(Alias::new(sum, "b"))));
+        let sent = 
Expression::new(ExpressionNode::Sent(Box::new(SentExpression {
+            expression: original.to_proto(),
+            rendered: original.render(),
+        })));
+        assert_eq!(sent.to_proto(), original.to_proto());
+        assert_eq!(sent.render(), "+(a, 1) AS b");
+        let over = Expression::function("abs", vec![sent]);
+        let over_original = Expression::function("abs", vec![original]);
+        assert_eq!(over.to_proto(), over_original.to_proto());
+        assert_eq!(over.render(), over_original.render());
+    }
+
     // Additional tests for expression render() methods and uncovered branches
     #[test]
     fn test_literal_render_byte() {
diff --git a/python/pyspark/sql/connect/_column_reduce.py 
b/python/pyspark/sql/connect/_column_reduce.py
new file mode 100644
index 00000000..44e727a2
--- /dev/null
+++ b/python/pyspark/sql/connect/_column_reduce.py
@@ -0,0 +1,59 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""The function that a pickled Column of the client unpickles with.
+
+``pyspark.sql.connect.column`` registers this module with cloudpickle to 
pickle by value, so a
+Column that a UDF or a ``foreachBatch`` function captures unpickles on a 
Python worker of the
+server, whose PySpark has no Rust client, while ``pickle`` pickles the 
function by reference, of
+this module, on the client.
+"""
+from typing import Any
+
+
+def column_of(expression: bytes, rendered: str) -> Any:
+    """The Column of ``expression``, the bytes of a 
``spark.connect.Expression``, and of its
+    rendering ``rendered``, which sends the expression as it is: of the Rust 
client, or, on a
+    Python worker of the server, PySpark Connect's Column of the expression, 
as PySpark Connect's
+    Column of its expression unpickles there, so everything that it calls 
there is of PySpark."""
+    try:
+        from pyspark import _pyspark
+    except ImportError:
+        from pyspark.sql.connect import proto
+        from pyspark.sql.connect.column import Column as ConnectColumn
+        from pyspark.sql.connect.expressions import Expression
+
+        # Without annotations: the class body looks its names up by name, 
which cloudpickle
+        # does not pickle with the function.
+        class Sent(Expression):
+            """An expression as it was sent, of its rendering."""
+
+            def __init__(self, expression, rendered):  # type: 
ignore[no-untyped-def]
+                super().__init__()
+                self._expression = expression
+                self._rendered = rendered
+
+            def to_plan(self, session):  # type: ignore[no-untyped-def]
+                sent = proto.Expression()
+                sent.CopyFrom(self._expression)
+                return sent
+
+            def __repr__(self):  # type: ignore[no-untyped-def]
+                return self._rendered
+
+        return ConnectColumn(Sent(proto.Expression.FromString(expression), 
rendered))
+
+    return _pyspark.Column._of_expression(expression, rendered)
diff --git a/python/pyspark/sql/connect/column.py 
b/python/pyspark/sql/connect/column.py
index a6c8d994..f589902f 100644
--- a/python/pyspark/sql/connect/column.py
+++ b/python/pyspark/sql/connect/column.py
@@ -15,5 +15,21 @@
 # limitations under the License.
 #
 """Rust-backed Column under the connect import path (pandas-on-Spark and 
connect tests import ConnectColumn from here)."""
+from typing import Any, Callable, Tuple
+
+from pyspark import cloudpickle as _cloudpickle
 from pyspark._pyspark import Column
+from pyspark.sql.connect import _column_reduce
+
 __all__ = ["Column"]
+
+_cloudpickle.register_pickle_by_value(_column_reduce)
+
+
+def _reduce_column(expression: bytes, rendered: str) -> Tuple[Callable[..., 
Any], Tuple[Any, ...]]:
+    """The ``__reduce__`` of a Column of the bytes of its 
``spark.connect.Expression`` and of its
+    rendering, as PySpark Connect's Column pickles its expression: the 
function of
+    ``pyspark.sql.connect._column_reduce``, which gives back the Column, which 
sends the
+    expression as it is. cloudpickle pickles that function by value, also to a 
Python worker of
+    the server, where it gives back PySpark Connect's Column, and ``pickle`` 
by reference."""
+    return _column_reduce.column_of, (expression, rendered)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 90ee3602..bc91ba74 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2994,6 +2994,27 @@ def 
test_foreach_batch_functions_that_capture_dataframes_run_as_pyspark(session,
         session.catalog.dropTempView(name)
 
 
+def test_foreach_batch_functions_that_capture_columns_run_as_pyspark(session, 
tmp_path):
+    # As PySpark's test_streaming_foreach_batch_external_column: a Column that 
a foreachBatch
+    # function captures unpickles on the Python worker of the server as 
PySpark Connect's, of
+    # the expression as it was sent.
+    (tmp_path / "input.txt").write_text("hello\nthis\n")
+    captured = F.upper(F.col("value"))
+    table = "foreach_batch_captured_column"
+
+    def func(df, batch_id):
+        
df.select(captured.alias("result")).write.mode("append").saveAsTable(table)
+
+    query = 
session.readStream.format("text").load(str(tmp_path)).writeStream.foreachBatch(func)
+    query = query.start()
+    try:
+        query.processAllAvailable()
+        assert sorted(row.result for row in session.table(table).collect()) == 
["HELLO", "THIS"]
+    finally:
+        query.stop()
+        session.sql(f"DROP TABLE IF EXISTS {table}").collect()
+
+
 def test_foreach_runs_a_function_and_a_writer_of_each_row_as_pyspark(session, 
tmp_path):
     # As PySpark's foreach: a function of a row, and an object of open, 
process and close, run
     # on each row of each batch, here writing a file of each row on the local 
server.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 05be95c1..0a4cef52 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -3961,6 +3961,35 @@ def test_decimal_type_methods():
     assert d.typeName() == "decimal"
 
 
+def test_columns_pickle_as_their_expressions_as_pyspark():
+    # As PySpark Connect's Columns, which pickle their expressions: pickle, 
cloudpickle and
+    # deepcopy give back a Column of the expression as it was sent, of its 
repr, and the
+    # expressions over it are those over the original.
+    import copy
+    from pyspark.serializers import CloudPickleSerializer
+    from pyspark.sql import functions as F
+
+    serializer = CloudPickleSerializer()
+    for column in [
+        F.col("a"),
+        (F.col("a") + 1).alias("b"),
+        F.lit(1.5),
+        F.when(F.col("a") > 1, "x").otherwise("y"),
+        F.struct("a", "b").getField("a"),
+        F.col("a").desc(),
+    ]:
+        for restored in (
+            pickle.loads(pickle.dumps(column)),
+            serializer.loads(serializer.dumps(column)),
+            copy.deepcopy(column),
+        ):
+            assert repr(restored) == repr(column)
+            assert restored.to_plan(None) == column.to_plan(None)
+            over, over_original = F.coalesce(restored, F.lit(0)), 
F.coalesce(column, F.lit(0))
+            assert repr(over) == repr(over_original)
+            assert over.to_plan(None) == over_original.to_plan(None)
+
+
 def test_column_repr_and_operations():
     """Test Column repr and various operations."""
     from pyspark.sql import functions as F


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to