This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 0d3b25ce [SPARK-61190][CONNECT][PYTHON] Keep the CASE WHEN of a 
pickled when() Column, so when and otherwise chain on it, in the Python client 
as PySpark does
0d3b25ce is described below

commit 0d3b25ce5b2c8752e7df0881dc091ddaae1f16d0
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 19:52:22 2026 +0900

    [SPARK-61190][CONNECT][PYTHON] Keep the CASE WHEN of a pickled when() 
Column, so when and otherwise chain on it, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61190](https://issues.apache.org/jira/browse/SPARK-61190): Keep the 
CASE WHEN of a pickled `when()` Column, so `when` and `otherwise` chain on it, 
as PySpark does.
    
    A `Column` pickles as its expression (SPARK-61167), so a `when()` column 
came back as an opaque expression, and `.when(...)` raised `INVALID_WHEN_USAGE` 
and `.otherwise(...)` raised "otherwise() can only be applied on a Column 
previously generated by when()". PySpark's `CaseWhen` keeps its branches and 
else value, so both chain.
    
    - `Column.__reduce__` also passes the (condition, value) Columns of the 
branches and the else Column of a CASE WHEN to `_reduce_column`.
    - The Rust `_of_expression` rebuilds a CASE WHEN of those branches, so an 
unpickled `when()` is a CASE WHEN again. Its plan is that of the chain built 
without the pickling.
    - On a Python worker of the server, `column_of` builds PySpark's `CaseWhen` 
(`_branches`, `_else_value`) of the same Columns, as PySpark does.
    
    ### Why are the changes needed?
    
    `pickle.loads(pickle.dumps(F.when(F.col("a") > 1, "big"))).when(F.col("a") 
> 0, "p")` raised `PySparkTypeError [INVALID_WHEN_USAGE]`, and 
`.otherwise("n")` raised, where official PySpark chains both. A `foreachBatch` 
function that captures a `when()` column and chains on it failed the same way 
on the server's worker.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `when` and `otherwise` chain on a pickled, cloudpickled or deep-copied 
`when()` Column, and on the one a `foreachBatch` function captures.
    
    ### How was this patch tested?
    
    - `test_pickled_when_columns_chain_as_pyspark` (offline, mock server): for 
pickle, cloudpickle and deepcopy, the chain `.when(...).otherwise(...)` has the 
same repr and sends the same expression as the chain built directly; 
`otherwise()` of a pickled `when()` matches; a `when()` after `otherwise()` 
raises `INVALID_WHEN_USAGE`. Fails without the fix.
    - `test_pickled_when_columns_chain_on_the_server_as_pyspark` (live): a 
pickled chain collects the same rows as the direct chain, and a `foreachBatch` 
function that captures a `when()` column and chains on it on the server writes 
the expected rows. Fails without the fix.
    - The chain cases (pickled, cloudpickled, deep-copied, `otherwise`, 
`otherwise` then `when`) give the same results and errors with official PySpark 
4.3.0. The foreachBatch capture of a `when()` column was checked against 
official with a zero-row stream only, so the live test is the one with rows.
    - The lib tests of both crates (599 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2228 passed, 8 skipped), and 
the live tests of when, pickling, foreachBatch, columns and UDFs (50 passed) 
against a local Spark 4.3.0 server; rustfmt passes, and clippy reports no 
diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #370 from HyukjinKwon/SPARK-61190-pickled-when-chains.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/column.rs              | 57 +++++++++++++++++++++++++++-
 python/pyspark/sql/connect/_column_reduce.py | 26 ++++++++++---
 python/pyspark/sql/connect/column.py         | 17 ++++++---
 python/tests/test_dropin_e2e.py              | 33 ++++++++++++++++
 python/tests/test_dropin_offline.py          | 36 ++++++++++++++++++
 5 files changed, 157 insertions(+), 12 deletions(-)

diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index 6617f730..bf563d52 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -691,18 +691,71 @@ impl PyColumn {
     /// `spark.connect.Expression` of this column and of its rendering, of 
which the function of
     /// `pyspark.sql.connect.column._reduce_column` gives back the Column, 
also on a Python worker
     /// of the server, such as that of a `foreachBatch` function, as PySpark 
Connect's Column.
+    /// A CASE WHEN also pickles its branches and else value, as PySpark's 
CaseWhen keeps them, so
+    /// that `when` and `otherwise` chain on the unpickled Column.
     fn __reduce__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
         use prost::Message;
         let expression = pyo3::types::PyBytes::new(py, 
&self.column.to_proto().encode_to_vec());
+        let column = |expr: &spark_connect::expression::Expression| -> 
PyResult<Py<PyColumn>> {
+            Py::new(py, PyColumn::new(Column::new(expr.clone())))
+        };
+        let (branches, else_value) = match self.column.expression().node() {
+            ExpressionNode::CaseWhen(case_when) => {
+                let branches = case_when
+                    .branches
+                    .iter()
+                    .map(|(condition, value)| Ok((column(condition)?, 
column(value)?)))
+                    .collect::<PyResult<Vec<_>>>()?;
+                let else_value = case_when
+                    .else_expr
+                    .as_ref()
+                    .map(|else_expr| column(else_expr))
+                    .transpose()?;
+                (Some(branches), else_value)
+            }
+            _ => (None, None),
+        };
         py.import("pyspark.sql.connect.column")?
             .getattr("_reduce_column")?
-            .call1((expression, self.column.expression().render()))
+            .call1((
+                expression,
+                self.column.expression().render(),
+                branches,
+                else_value,
+            ))
     }
 
     /// The Column of `expression`, the bytes of a `spark.connect.Expression`, 
and of its
     /// `rendered` rendering, as a pickled Column unpickles: it sends the 
expression as it is.
+    /// The CASE WHEN of `branches` and `else_value`, the Columns of its 
parts, is built as it was.
     #[staticmethod]
-    fn _of_expression(expression: &[u8], rendered: String) -> 
PyResult<PyColumn> {
+    #[pyo3(signature = (expression, rendered, branches=None, else_value=None))]
+    fn _of_expression(
+        expression: &[u8],
+        rendered: String,
+        branches: Option<Vec<(PyRef<'_, PyColumn>, PyRef<'_, PyColumn>)>>,
+        else_value: Option<PyRef<'_, PyColumn>>,
+    ) -> PyResult<PyColumn> {
+        if let Some(branches) = branches {
+            let branches = branches
+                .iter()
+                .map(|(condition, value)| {
+                    (
+                        condition.column.expression().clone(),
+                        value.column.expression().clone(),
+                    )
+                })
+                .collect();
+            let mut case_when = 
spark_connect::expression::CaseWhen::new(branches);
+            if let Some(else_value) = else_value {
+                case_when = 
case_when.with_else(else_value.column.expression().clone());
+            }
+            return Ok(PyColumn::new(Column::new(
+                
spark_connect::expression::Expression::new(ExpressionNode::CaseWhen(Box::new(
+                    case_when,
+                ))),
+            )));
+        }
         use prost::Message;
         let expression = spark_connect_proto::Expression::decode(expression)
             .map_err(|error| 
pyo3::exceptions::PyValueError::new_err(error.to_string()))?;
diff --git a/python/pyspark/sql/connect/_column_reduce.py 
b/python/pyspark/sql/connect/_column_reduce.py
index 44e727a2..ff047da2 100644
--- a/python/pyspark/sql/connect/_column_reduce.py
+++ b/python/pyspark/sql/connect/_column_reduce.py
@@ -21,20 +21,36 @@ Column that a UDF or a ``foreachBatch`` function captures 
unpickles on a Python
 server, whose PySpark has no Rust client, while ``pickle`` pickles the 
function by reference, of
 this module, on the client.
 """
-from typing import Any
+from typing import Any, List, Optional, Tuple
 
 
-def column_of(expression: bytes, rendered: str) -> Any:
+def column_of(
+    expression: bytes,
+    rendered: str,
+    branches: Optional[List[Tuple[Any, Any]]] = None,
+    else_value: Any = None,
+) -> Any:
     """The Column of ``expression``, the bytes of a 
``spark.connect.Expression``, and of its
     rendering ``rendered``, which sends the expression as it is: of the Rust 
client, or, on a
     Python worker of the server, PySpark Connect's Column of the expression, 
as PySpark Connect's
-    Column of its expression unpickles there, so everything that it calls 
there is of PySpark."""
+    Column of its expression unpickles there, so everything that it calls 
there is of PySpark.
+    For a CASE WHEN, ``branches`` are the (condition, value) Columns of its 
branches and
+    ``else_value`` the Column of its else value, so that it is a CASE WHEN 
again."""
     try:
         from pyspark import _pyspark
     except ImportError:
         from pyspark.sql.connect import proto
         from pyspark.sql.connect.column import Column as ConnectColumn
-        from pyspark.sql.connect.expressions import Expression
+        from pyspark.sql.connect.expressions import CaseWhen, Expression
+
+        if branches is not None:
+            # As PySpark's CaseWhen of its branches and else value.
+            return ConnectColumn(
+                CaseWhen(
+                    [(condition._expr, value._expr) for condition, value in 
branches],
+                    None if else_value is None else else_value._expr,
+                )
+            )
 
         # Without annotations: the class body looks its names up by name, 
which cloudpickle
         # does not pickle with the function.
@@ -56,4 +72,4 @@ def column_of(expression: bytes, rendered: str) -> Any:
 
         return ConnectColumn(Sent(proto.Expression.FromString(expression), 
rendered))
 
-    return _pyspark.Column._of_expression(expression, rendered)
+    return _pyspark.Column._of_expression(expression, rendered, branches, 
else_value)
diff --git a/python/pyspark/sql/connect/column.py 
b/python/pyspark/sql/connect/column.py
index f589902f..d40afba0 100644
--- a/python/pyspark/sql/connect/column.py
+++ b/python/pyspark/sql/connect/column.py
@@ -15,7 +15,7 @@
 # limitations under the License.
 #
 """Rust-backed Column under the connect import path (pandas-on-Spark and 
connect tests import ConnectColumn from here)."""
-from typing import Any, Callable, Tuple
+from typing import Any, Callable, List, Optional, Tuple
 
 from pyspark import cloudpickle as _cloudpickle
 from pyspark._pyspark import Column
@@ -26,10 +26,17 @@ __all__ = ["Column"]
 _cloudpickle.register_pickle_by_value(_column_reduce)
 
 
-def _reduce_column(expression: bytes, rendered: str) -> Tuple[Callable[..., 
Any], Tuple[Any, ...]]:
+def _reduce_column(
+    expression: bytes,
+    rendered: str,
+    branches: Optional[List[Tuple[Any, Any]]] = None,
+    else_value: Any = None,
+) -> Tuple[Callable[..., Any], Tuple[Any, ...]]:
     """The ``__reduce__`` of a Column of the bytes of its 
``spark.connect.Expression`` and of its
     rendering, as PySpark Connect's Column pickles its expression: the 
function of
     ``pyspark.sql.connect._column_reduce``, which gives back the Column, which 
sends the
-    expression as it is. cloudpickle pickles that function by value, also to a 
Python worker of
-    the server, where it gives back PySpark Connect's Column, and ``pickle`` 
by reference."""
-    return _column_reduce.column_of, (expression, rendered)
+    expression as it is. A CASE WHEN gives its branches and else value too, as 
Columns, so the
+    Column gives back a CASE WHEN. cloudpickle pickles that function by value, 
also to a Python
+    worker of the server, where it gives back PySpark Connect's Column, and 
``pickle`` by
+    reference."""
+    return _column_reduce.column_of, (expression, rendered, branches, 
else_value)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 65b4a85e..16fdf0c6 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -875,6 +875,39 @@ ARROW_TABLE_ROWS = [
 ]
 
 
+def test_pickled_when_columns_chain_on_the_server_as_pyspark(session, 
tmp_path):
+    # As PySpark's: a pickled when() Column keeps its CASE WHEN, so it 
collects as the chain
+    # built without the pickling, and a foreachBatch function that captures it 
chains on it on
+    # the Python worker of the server.
+    import pickle
+
+    (tmp_path / "input.txt").write_text("hello\nthis\nother\n")
+    df = session.createDataFrame([(1,), (2,), (3,)], "a int")
+    chain = F.when(F.col("a") > 2, "big").when(F.col("a") > 1, 
"mid").otherwise("small")
+    assert df.select(pickle.loads(pickle.dumps(chain)).alias("r")).collect() 
== (
+        df.select(chain.alias("r")).collect()
+    )
+    assert [row.r for row in 
df.select(pickle.loads(pickle.dumps(chain)).alias("r")).collect()] == [
+        "small", "mid", "big"]
+
+    captured = F.when(F.col("value") == "hello", "h")
+    table = "foreach_batch_captured_when"
+
+    def func(batch, batch_id):
+        batch.select(
+            captured.when(F.col("value") == "this", 
"t").otherwise("x").alias("result")
+        ).write.mode("append").saveAsTable(table)
+
+    query = 
session.readStream.format("text").load(str(tmp_path)).writeStream.foreachBatch(func)
+    query = query.start()
+    try:
+        query.processAllAvailable()
+        assert sorted(row.result for row in session.table(table).collect()) == 
["h", "t", "x"]
+    finally:
+        query.stop()
+        session.sql(f"DROP TABLE IF EXISTS {table}").collect()
+
+
 @pytest.mark.parametrize(
     "schema, expected",
     [
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 1d5f3d80..a4e4221d 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1350,6 +1350,42 @@ def 
test_sql_utils_reexports_the_exceptions_of_pyspark(mock_connect):
     assert "TABLE_OR_VIEW_NOT_FOUND" in str(error.value)
 
 
+def test_pickled_when_columns_chain_as_pyspark(mock_connect):
+    # As PySpark's CaseWhen: a pickled, cloudpickled or deep-copied when() 
Column keeps its CASE
+    # WHEN, so when() and otherwise() chain on it, and the chain sends the 
plan of the chain
+    # built without the pickling.
+    import copy
+
+    import cloudpickle
+    from pyspark.errors import PySparkTypeError
+    from pyspark.sql import functions as F
+
+    spark, service = mock_connect
+    df = spark.range(1)
+
+    def sent_expression(dataframe):
+        # ExecutePlanRequest.plan=3, Plan.root=1, Relation.project=3, 
Project.expressions=3.
+        dataframe.collect()
+        plan = 
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+        return parse_message(parse_message(parse_message(plan)[3][0])[3][0])
+
+    def chain(when):
+        return when.when(F.col("id") > 5, "huge").otherwise("small")
+
+    direct = F.when(F.col("id") > 0, "big")
+    expected = sent_expression(df.select(chain(direct).alias("x")))
+    for copied in (pickle.loads(pickle.dumps(direct)),
+                   cloudpickle.loads(cloudpickle.dumps(direct)),
+                   copy.deepcopy(direct)):
+        assert repr(chain(copied)) == repr(chain(direct))
+        assert sent_expression(df.select(chain(copied).alias("x"))) == expected
+        # otherwise() of a pickled when(), and a when() after the chain, which 
PySpark rejects.
+        assert repr(copied.otherwise("n")) == repr(direct.otherwise("n"))
+        with pytest.raises(PySparkTypeError) as error:
+            copied.otherwise("n").when(F.col("id") > 0, "p")
+        assert error.value.getCondition() == "INVALID_WHEN_USAGE"
+
+
 def test_functions_build_columns():
     from pyspark.sql import functions as F
     c = F.col("id")


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to