This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 0d3b25ce [SPARK-61190][CONNECT][PYTHON] Keep the CASE WHEN of a
pickled when() Column, so when and otherwise chain on it, in the Python client
as PySpark does
0d3b25ce is described below
commit 0d3b25ce5b2c8752e7df0881dc091ddaae1f16d0
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 19:52:22 2026 +0900
[SPARK-61190][CONNECT][PYTHON] Keep the CASE WHEN of a pickled when()
Column, so when and otherwise chain on it, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61190](https://issues.apache.org/jira/browse/SPARK-61190): Keep the
CASE WHEN of a pickled `when()` Column, so `when` and `otherwise` chain on it,
as PySpark does.
A `Column` pickles as its expression (SPARK-61167), so a `when()` column
came back as an opaque expression, and `.when(...)` raised `INVALID_WHEN_USAGE`
and `.otherwise(...)` raised "otherwise() can only be applied on a Column
previously generated by when()". PySpark's `CaseWhen` keeps its branches and
else value, so both chain.
- `Column.__reduce__` also passes the (condition, value) Columns of the
branches and the else Column of a CASE WHEN to `_reduce_column`.
- The Rust `_of_expression` rebuilds a CASE WHEN of those branches, so an
unpickled `when()` is a CASE WHEN again. Its plan is that of the chain built
without the pickling.
- On a Python worker of the server, `column_of` builds PySpark's `CaseWhen`
(`_branches`, `_else_value`) of the same Columns, as PySpark does.
### Why are the changes needed?
`pickle.loads(pickle.dumps(F.when(F.col("a") > 1, "big"))).when(F.col("a")
> 0, "p")` raised `PySparkTypeError [INVALID_WHEN_USAGE]`, and
`.otherwise("n")` raised, where official PySpark chains both. A `foreachBatch`
function that captures a `when()` column and chains on it failed the same way
on the server's worker.
### Does this PR introduce _any_ user-facing change?
Yes. `when` and `otherwise` chain on a pickled, cloudpickled or deep-copied
`when()` Column, and on the one a `foreachBatch` function captures.
### How was this patch tested?
- `test_pickled_when_columns_chain_as_pyspark` (offline, mock server): for
pickle, cloudpickle and deepcopy, the chain `.when(...).otherwise(...)` has the
same repr and sends the same expression as the chain built directly;
`otherwise()` of a pickled `when()` matches; a `when()` after `otherwise()`
raises `INVALID_WHEN_USAGE`. Fails without the fix.
- `test_pickled_when_columns_chain_on_the_server_as_pyspark` (live): a
pickled chain collects the same rows as the direct chain, and a `foreachBatch`
function that captures a `when()` column and chains on it on the server writes
the expected rows. Fails without the fix.
- The chain cases (pickled, cloudpickled, deep-copied, `otherwise`,
`otherwise` then `when`) give the same results and errors with official PySpark
4.3.0. The foreachBatch capture of a `when()` column was checked against
official with a zero-row stream only, so the live test is the one with rows.
- The lib tests of both crates (599 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2228 passed, 8 skipped), and
the live tests of when, pickling, foreachBatch, columns and UDFs (50 passed)
against a local Spark 4.3.0 server; rustfmt passes, and clippy reports no
diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #370 from HyukjinKwon/SPARK-61190-pickled-when-chains.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/column.rs | 57 +++++++++++++++++++++++++++-
python/pyspark/sql/connect/_column_reduce.py | 26 ++++++++++---
python/pyspark/sql/connect/column.py | 17 ++++++---
python/tests/test_dropin_e2e.py | 33 ++++++++++++++++
python/tests/test_dropin_offline.py | 36 ++++++++++++++++++
5 files changed, 157 insertions(+), 12 deletions(-)
diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index 6617f730..bf563d52 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -691,18 +691,71 @@ impl PyColumn {
/// `spark.connect.Expression` of this column and of its rendering, of
which the function of
/// `pyspark.sql.connect.column._reduce_column` gives back the Column,
also on a Python worker
/// of the server, such as that of a `foreachBatch` function, as PySpark
Connect's Column.
+ /// A CASE WHEN also pickles its branches and else value, as PySpark's
CaseWhen keeps them, so
+ /// that `when` and `otherwise` chain on the unpickled Column.
fn __reduce__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
use prost::Message;
let expression = pyo3::types::PyBytes::new(py,
&self.column.to_proto().encode_to_vec());
+ let column = |expr: &spark_connect::expression::Expression| ->
PyResult<Py<PyColumn>> {
+ Py::new(py, PyColumn::new(Column::new(expr.clone())))
+ };
+ let (branches, else_value) = match self.column.expression().node() {
+ ExpressionNode::CaseWhen(case_when) => {
+ let branches = case_when
+ .branches
+ .iter()
+ .map(|(condition, value)| Ok((column(condition)?,
column(value)?)))
+ .collect::<PyResult<Vec<_>>>()?;
+ let else_value = case_when
+ .else_expr
+ .as_ref()
+ .map(|else_expr| column(else_expr))
+ .transpose()?;
+ (Some(branches), else_value)
+ }
+ _ => (None, None),
+ };
py.import("pyspark.sql.connect.column")?
.getattr("_reduce_column")?
- .call1((expression, self.column.expression().render()))
+ .call1((
+ expression,
+ self.column.expression().render(),
+ branches,
+ else_value,
+ ))
}
/// The Column of `expression`, the bytes of a `spark.connect.Expression`,
and of its
/// `rendered` rendering, as a pickled Column unpickles: it sends the
expression as it is.
+ /// The CASE WHEN of `branches` and `else_value`, the Columns of its
parts, is built as it was.
#[staticmethod]
- fn _of_expression(expression: &[u8], rendered: String) ->
PyResult<PyColumn> {
+ #[pyo3(signature = (expression, rendered, branches=None, else_value=None))]
+ fn _of_expression(
+ expression: &[u8],
+ rendered: String,
+ branches: Option<Vec<(PyRef<'_, PyColumn>, PyRef<'_, PyColumn>)>>,
+ else_value: Option<PyRef<'_, PyColumn>>,
+ ) -> PyResult<PyColumn> {
+ if let Some(branches) = branches {
+ let branches = branches
+ .iter()
+ .map(|(condition, value)| {
+ (
+ condition.column.expression().clone(),
+ value.column.expression().clone(),
+ )
+ })
+ .collect();
+ let mut case_when =
spark_connect::expression::CaseWhen::new(branches);
+ if let Some(else_value) = else_value {
+ case_when =
case_when.with_else(else_value.column.expression().clone());
+ }
+ return Ok(PyColumn::new(Column::new(
+
spark_connect::expression::Expression::new(ExpressionNode::CaseWhen(Box::new(
+ case_when,
+ ))),
+ )));
+ }
use prost::Message;
let expression = spark_connect_proto::Expression::decode(expression)
.map_err(|error|
pyo3::exceptions::PyValueError::new_err(error.to_string()))?;
diff --git a/python/pyspark/sql/connect/_column_reduce.py
b/python/pyspark/sql/connect/_column_reduce.py
index 44e727a2..ff047da2 100644
--- a/python/pyspark/sql/connect/_column_reduce.py
+++ b/python/pyspark/sql/connect/_column_reduce.py
@@ -21,20 +21,36 @@ Column that a UDF or a ``foreachBatch`` function captures
unpickles on a Python
server, whose PySpark has no Rust client, while ``pickle`` pickles the
function by reference, of
this module, on the client.
"""
-from typing import Any
+from typing import Any, List, Optional, Tuple
-def column_of(expression: bytes, rendered: str) -> Any:
+def column_of(
+ expression: bytes,
+ rendered: str,
+ branches: Optional[List[Tuple[Any, Any]]] = None,
+ else_value: Any = None,
+) -> Any:
"""The Column of ``expression``, the bytes of a
``spark.connect.Expression``, and of its
rendering ``rendered``, which sends the expression as it is: of the Rust
client, or, on a
Python worker of the server, PySpark Connect's Column of the expression,
as PySpark Connect's
- Column of its expression unpickles there, so everything that it calls
there is of PySpark."""
+ Column of its expression unpickles there, so everything that it calls
there is of PySpark.
+ For a CASE WHEN, ``branches`` are the (condition, value) Columns of its
branches and
+ ``else_value`` the Column of its else value, so that it is a CASE WHEN
again."""
try:
from pyspark import _pyspark
except ImportError:
from pyspark.sql.connect import proto
from pyspark.sql.connect.column import Column as ConnectColumn
- from pyspark.sql.connect.expressions import Expression
+ from pyspark.sql.connect.expressions import CaseWhen, Expression
+
+ if branches is not None:
+ # As PySpark's CaseWhen of its branches and else value.
+ return ConnectColumn(
+ CaseWhen(
+ [(condition._expr, value._expr) for condition, value in
branches],
+ None if else_value is None else else_value._expr,
+ )
+ )
# Without annotations: the class body looks its names up by name,
which cloudpickle
# does not pickle with the function.
@@ -56,4 +72,4 @@ def column_of(expression: bytes, rendered: str) -> Any:
return ConnectColumn(Sent(proto.Expression.FromString(expression),
rendered))
- return _pyspark.Column._of_expression(expression, rendered)
+ return _pyspark.Column._of_expression(expression, rendered, branches,
else_value)
diff --git a/python/pyspark/sql/connect/column.py
b/python/pyspark/sql/connect/column.py
index f589902f..d40afba0 100644
--- a/python/pyspark/sql/connect/column.py
+++ b/python/pyspark/sql/connect/column.py
@@ -15,7 +15,7 @@
# limitations under the License.
#
"""Rust-backed Column under the connect import path (pandas-on-Spark and
connect tests import ConnectColumn from here)."""
-from typing import Any, Callable, Tuple
+from typing import Any, Callable, List, Optional, Tuple
from pyspark import cloudpickle as _cloudpickle
from pyspark._pyspark import Column
@@ -26,10 +26,17 @@ __all__ = ["Column"]
_cloudpickle.register_pickle_by_value(_column_reduce)
-def _reduce_column(expression: bytes, rendered: str) -> Tuple[Callable[...,
Any], Tuple[Any, ...]]:
+def _reduce_column(
+ expression: bytes,
+ rendered: str,
+ branches: Optional[List[Tuple[Any, Any]]] = None,
+ else_value: Any = None,
+) -> Tuple[Callable[..., Any], Tuple[Any, ...]]:
"""The ``__reduce__`` of a Column of the bytes of its
``spark.connect.Expression`` and of its
rendering, as PySpark Connect's Column pickles its expression: the
function of
``pyspark.sql.connect._column_reduce``, which gives back the Column, which
sends the
- expression as it is. cloudpickle pickles that function by value, also to a
Python worker of
- the server, where it gives back PySpark Connect's Column, and ``pickle``
by reference."""
- return _column_reduce.column_of, (expression, rendered)
+ expression as it is. A CASE WHEN gives its branches and else value too, as
Columns, so the
+ Column gives back a CASE WHEN. cloudpickle pickles that function by value,
also to a Python
+ worker of the server, where it gives back PySpark Connect's Column, and
``pickle`` by
+ reference."""
+ return _column_reduce.column_of, (expression, rendered, branches,
else_value)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 65b4a85e..16fdf0c6 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -875,6 +875,39 @@ ARROW_TABLE_ROWS = [
]
+def test_pickled_when_columns_chain_on_the_server_as_pyspark(session,
tmp_path):
+ # As PySpark's: a pickled when() Column keeps its CASE WHEN, so it
collects as the chain
+ # built without the pickling, and a foreachBatch function that captures it
chains on it on
+ # the Python worker of the server.
+ import pickle
+
+ (tmp_path / "input.txt").write_text("hello\nthis\nother\n")
+ df = session.createDataFrame([(1,), (2,), (3,)], "a int")
+ chain = F.when(F.col("a") > 2, "big").when(F.col("a") > 1,
"mid").otherwise("small")
+ assert df.select(pickle.loads(pickle.dumps(chain)).alias("r")).collect()
== (
+ df.select(chain.alias("r")).collect()
+ )
+ assert [row.r for row in
df.select(pickle.loads(pickle.dumps(chain)).alias("r")).collect()] == [
+ "small", "mid", "big"]
+
+ captured = F.when(F.col("value") == "hello", "h")
+ table = "foreach_batch_captured_when"
+
+ def func(batch, batch_id):
+ batch.select(
+ captured.when(F.col("value") == "this",
"t").otherwise("x").alias("result")
+ ).write.mode("append").saveAsTable(table)
+
+ query =
session.readStream.format("text").load(str(tmp_path)).writeStream.foreachBatch(func)
+ query = query.start()
+ try:
+ query.processAllAvailable()
+ assert sorted(row.result for row in session.table(table).collect()) ==
["h", "t", "x"]
+ finally:
+ query.stop()
+ session.sql(f"DROP TABLE IF EXISTS {table}").collect()
+
+
@pytest.mark.parametrize(
"schema, expected",
[
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 1d5f3d80..a4e4221d 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1350,6 +1350,42 @@ def
test_sql_utils_reexports_the_exceptions_of_pyspark(mock_connect):
assert "TABLE_OR_VIEW_NOT_FOUND" in str(error.value)
+def test_pickled_when_columns_chain_as_pyspark(mock_connect):
+ # As PySpark's CaseWhen: a pickled, cloudpickled or deep-copied when()
Column keeps its CASE
+ # WHEN, so when() and otherwise() chain on it, and the chain sends the
plan of the chain
+ # built without the pickling.
+ import copy
+
+ import cloudpickle
+ from pyspark.errors import PySparkTypeError
+ from pyspark.sql import functions as F
+
+ spark, service = mock_connect
+ df = spark.range(1)
+
+ def sent_expression(dataframe):
+ # ExecutePlanRequest.plan=3, Plan.root=1, Relation.project=3,
Project.expressions=3.
+ dataframe.collect()
+ plan =
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+ return parse_message(parse_message(parse_message(plan)[3][0])[3][0])
+
+ def chain(when):
+ return when.when(F.col("id") > 5, "huge").otherwise("small")
+
+ direct = F.when(F.col("id") > 0, "big")
+ expected = sent_expression(df.select(chain(direct).alias("x")))
+ for copied in (pickle.loads(pickle.dumps(direct)),
+ cloudpickle.loads(cloudpickle.dumps(direct)),
+ copy.deepcopy(direct)):
+ assert repr(chain(copied)) == repr(chain(direct))
+ assert sent_expression(df.select(chain(copied).alias("x"))) == expected
+ # otherwise() of a pickled when(), and a when() after the chain, which
PySpark rejects.
+ assert repr(copied.otherwise("n")) == repr(direct.otherwise("n"))
+ with pytest.raises(PySparkTypeError) as error:
+ copied.otherwise("n").when(F.col("id") > 0, "p")
+ assert error.value.getCondition() == "INVALID_WHEN_USAGE"
+
+
def test_functions_build_columns():
from pyspark.sql import functions as F
c = F.col("id")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]