This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 97945273 [SPARK-61199][CONNECT][PYTHON] Keep the sort order of a 
pickled sort Column, so that orderBy, Window.orderBy and repartitionByRange 
sort by it, in the Python client as PySpark does
97945273 is described below

commit 9794527370c6efc4dd99a4eaaa508764bb9de1b0
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 09:07:52 2026 +0900

    [SPARK-61199][CONNECT][PYTHON] Keep the sort order of a pickled sort 
Column, so that orderBy, Window.orderBy and repartitionByRange sort by it, in 
the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61199](https://issues.apache.org/jira/browse/SPARK-61199): PySpark 
Connect's Column pickles its expression, and a sort Column's expression is a 
`SortOrder` of its child, its direction and its null ordering. So a pickled 
`F.col("x").desc()` is a sort order again wherever it unpickles, and `orderBy`, 
`Window.orderBy` and `repartitionByRange` sort by it.
    
    A Column of this client pickles as the bytes of its expression 
(SPARK-61167), which unpickle as an opaque expression. Now a sort order pickles 
as its child Column and the name of its sort method (`asc_nulls_first`, 
`asc_nulls_last`, `desc_nulls_first` or `desc_nulls_last`), as SPARK-61190 did 
for CASE WHEN:
    
    - `Column.__reduce__` (Rust) passes `(child, method)` of a sort order to 
`_reduce_column`.
    - `pyspark.sql.connect._column_reduce.column_of` calls that method on the 
unpickled child. On the client the child is a Column of this client. On a 
Python worker of the server it is PySpark Connect's Column, whose method gives 
back PySpark's `SortOrder`.
    
    ### Why are the changes needed?
    
    On a Python worker of the server, PySpark's `DataFrame.orderBy` wraps a 
Column that is not a `SortOrder` in `asc()`. So a `foreachBatch` function that 
captured a sort Column sent a sort order of a sort order:
    
    ```
    captured = F.col("value").desc()
    def func(batch, batch_id):
        batch.orderBy(captured).limit(1).write.mode("append").saveAsTable("t")
    # SparkException: [INTERNAL_ERROR] Cannot evaluate expression: ... DESC 
NULLS LAST
    ```
    
    On the client, `Window.orderBy` and `repartitionByRange` wrapped an 
unpickled sort Column in ASC NULLS FIRST in the same way:
    
    ```
    >>> Window.orderBy(pickle.loads(pickle.dumps(F.col("x").desc())))
    WindowSpec(OrderBy(x DESC NULLS LAST ASC NULLS FIRST))     # official: 
WindowSpec(OrderBy(x DESC NULLS LAST))
    >>> df.select(F.row_number().over(that_window)).collect()   # a partition 
of two or more rows
    SparkException: [INTERNAL_ERROR] Cannot evaluate expression: input[1, int, 
true] DESC NULLS LAST
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A pickled, cloudpickled or deep-copied sort Column is a sort order: 
`orderBy`, `Window.orderBy` and `repartitionByRange` sort by it on the client, 
and so do the DataFrame methods on the Python worker of the server, for a sort 
Column that a `foreachBatch` function or a UDF captures.
    
    ### How was this patch tested?
    
    - `test_pickled_sort_columns_are_sort_orders_as_pyspark` (offline, mock 
server): for `asc`, `desc`, `asc_nulls_last`, `desc_nulls_first` and `F.desc`, 
each pickled, cloudpickled and deep-copied, the Column and its `Window.orderBy` 
have the reprs of the Column built without the pickling. `orderBy`, 
`repartitionByRange` and a window over it also send the same sort orders. A 
sort order of an expression, pickled twice, matches too. It fails without the 
change, where the `Window.orderBy` o [...]
    - `test_pickled_sort_columns_sort_on_the_server_as_pyspark` (live): 
`orderBy`, `row_number().over(Window.orderBy(...))` and `repartitionByRange` of 
a pickled `desc` and `asc_nulls_last` give the rows of the direct Column. A 
`foreachBatch` function that captures `F.col("value").desc()` and sorts by it 
on the server writes the largest value. It fails without the change, with the 
INTERNAL_ERROR above.
    - The same cases with official PySpark 4.3.0 against the same server, for 
`desc`, `asc_nulls_last`, `desc_nulls_first` and `asc`: the reprs, `orderBy`, 
windows and `repartitionByRange` give identical output.
    - The pickling tests of the offline suite (`-k pickl`, 67 passed) and the 
pickled `when()` live test.
    - The lib tests of both crates (618 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2457 passed, 8 skipped), and the 
live drop-in, Arrow and ML suites (215 passed) and live cases (11 passed) 
against a local Spark 4.3.0 server. Three live foreachBatch and 
progress-handler tests failed with `Python worker failed to connect back` while 
the machine was overloaded, and passed when run again. Clippy reports no 
diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #388 from HyukjinKwon/SPARK-61199-pickled-sort-orders.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/column.rs              | 19 +++++++++--
 python/pyspark/sql/connect/_column_reduce.py | 10 +++++-
 python/pyspark/sql/connect/column.py         |  8 +++--
 python/tests/test_dropin_e2e.py              | 37 +++++++++++++++++++++
 python/tests/test_dropin_offline.py          | 48 ++++++++++++++++++++++++++++
 5 files changed, 116 insertions(+), 6 deletions(-)

diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index bf563d52..08306e63 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -3,7 +3,7 @@
 use pyo3::prelude::*;
 use pyo3::types::PyDict;
 use spark_connect::column::Column;
-use spark_connect::expression::{Alias, ExpressionNode};
+use spark_connect::expression::{Alias, ExpressionNode, NullOrdering};
 
 use crate::functions::to_column;
 
@@ -692,7 +692,9 @@ impl PyColumn {
     /// `pyspark.sql.connect.column._reduce_column` gives back the Column, 
also on a Python worker
     /// of the server, such as that of a `foreachBatch` function, as PySpark 
Connect's Column.
     /// A CASE WHEN also pickles its branches and else value, as PySpark's 
CaseWhen keeps them, so
-    /// that `when` and `otherwise` chain on the unpickled Column.
+    /// that `when` and `otherwise` chain on the unpickled Column. A sort 
order pickles its child
+    /// Column and its sort method, as PySpark's SortOrder keeps its child and 
direction, so that
+    /// the unpickled Column is a sort order of `orderBy`, `Window.orderBy` 
and the others.
     fn __reduce__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
         use prost::Message;
         let expression = pyo3::types::PyBytes::new(py, 
&self.column.to_proto().encode_to_vec());
@@ -715,6 +717,18 @@ impl PyColumn {
             }
             _ => (None, None),
         };
+        let sort = match self.column.expression().node() {
+            ExpressionNode::SortOrder(sort) => {
+                let method = match (sort.ascending, sort.null_ordering) {
+                    (true, NullOrdering::First) => "asc_nulls_first",
+                    (true, NullOrdering::Last) => "asc_nulls_last",
+                    (false, NullOrdering::First) => "desc_nulls_first",
+                    (false, NullOrdering::Last) => "desc_nulls_last",
+                };
+                Some((column(&sort.child)?, method))
+            }
+            _ => None,
+        };
         py.import("pyspark.sql.connect.column")?
             .getattr("_reduce_column")?
             .call1((
@@ -722,6 +736,7 @@ impl PyColumn {
                 self.column.expression().render(),
                 branches,
                 else_value,
+                sort,
             ))
     }
 
diff --git a/python/pyspark/sql/connect/_column_reduce.py 
b/python/pyspark/sql/connect/_column_reduce.py
index ff047da2..d9a6cfbe 100644
--- a/python/pyspark/sql/connect/_column_reduce.py
+++ b/python/pyspark/sql/connect/_column_reduce.py
@@ -29,13 +29,21 @@ def column_of(
     rendered: str,
     branches: Optional[List[Tuple[Any, Any]]] = None,
     else_value: Any = None,
+    sort: Optional[Tuple[Any, str]] = None,
 ) -> Any:
     """The Column of ``expression``, the bytes of a 
``spark.connect.Expression``, and of its
     rendering ``rendered``, which sends the expression as it is: of the Rust 
client, or, on a
     Python worker of the server, PySpark Connect's Column of the expression, 
as PySpark Connect's
     Column of its expression unpickles there, so everything that it calls 
there is of PySpark.
     For a CASE WHEN, ``branches`` are the (condition, value) Columns of its 
branches and
-    ``else_value`` the Column of its else value, so that it is a CASE WHEN 
again."""
+    ``else_value`` the Column of its else value, so that it is a CASE WHEN 
again. For a sort
+    order, ``sort`` is its child Column and the name of its sort method, such 
as
+    ``desc_nulls_last``, so that it is a sort order again."""
+    if sort is not None:
+        # The child is a Column of the client, or PySpark Connect's on a 
worker, whose sort
+        # method gives back the sort order, as PySpark's SortOrder of its 
child.
+        child, method = sort
+        return getattr(child, method)()
     try:
         from pyspark import _pyspark
     except ImportError:
diff --git a/python/pyspark/sql/connect/column.py 
b/python/pyspark/sql/connect/column.py
index d40afba0..8d4759e0 100644
--- a/python/pyspark/sql/connect/column.py
+++ b/python/pyspark/sql/connect/column.py
@@ -31,12 +31,14 @@ def _reduce_column(
     rendered: str,
     branches: Optional[List[Tuple[Any, Any]]] = None,
     else_value: Any = None,
+    sort: Optional[Tuple[Any, str]] = None,
 ) -> Tuple[Callable[..., Any], Tuple[Any, ...]]:
     """The ``__reduce__`` of a Column of the bytes of its 
``spark.connect.Expression`` and of its
     rendering, as PySpark Connect's Column pickles its expression: the 
function of
     ``pyspark.sql.connect._column_reduce``, which gives back the Column, which 
sends the
     expression as it is. A CASE WHEN gives its branches and else value too, as 
Columns, so the
-    Column gives back a CASE WHEN. cloudpickle pickles that function by value, 
also to a Python
-    worker of the server, where it gives back PySpark Connect's Column, and 
``pickle`` by
+    Column gives back a CASE WHEN. A sort order gives its child Column and its 
sort method, so
+    the Column gives back a sort order. cloudpickle pickles that function by 
value, also to a
+    Python worker of the server, where it gives back PySpark Connect's Column, 
and ``pickle`` by
     reference."""
-    return _column_reduce.column_of, (expression, rendered, branches, 
else_value)
+    return _column_reduce.column_of, (expression, rendered, branches, 
else_value, sort)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index efddc6bc..7bc7a6b7 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -935,6 +935,43 @@ ARROW_TABLE_ROWS = [
 ]
 
 
+def test_pickled_sort_columns_sort_on_the_server_as_pyspark(session, tmp_path):
+    # As PySpark's: a pickled sort Column is a sort order again, so orderBy, a 
window and
+    # repartitionByRange sort by it as by the Column built without the 
pickling, and a
+    # foreachBatch function that captures it sorts by it on the Python worker 
of the server.
+    import pickle
+
+    from pyspark.sql.window import Window
+
+    df = session.createDataFrame([(1, "a"), (3, "b"), (None, "c"), (2, "d")], 
"x int, y string")
+    for direct in (F.col("x").desc(), F.col("x").asc_nulls_last()):
+        copied = pickle.loads(pickle.dumps(direct))
+        assert df.orderBy(copied).collect() == df.orderBy(direct).collect()
+        ranked = df.select("y", 
F.row_number().over(Window.orderBy(copied)).alias("r"))
+        assert sorted(ranked.collect()) == sorted(
+            df.select("y", 
F.row_number().over(Window.orderBy(direct)).alias("r")).collect()
+        )
+        by_y = lambda row: row.y  # noqa: E731
+        assert sorted(df.repartitionByRange(2, copied).collect(), key=by_y) == 
sorted(
+            df.collect(), key=by_y)
+
+    (tmp_path / "input.txt").write_text("a\nc\nb\n")
+    captured = F.col("value").desc()
+    table = "foreach_batch_captured_sort"
+
+    def func(batch, batch_id):
+        
batch.orderBy(captured).limit(1).write.mode("append").saveAsTable(table)
+
+    query = 
session.readStream.format("text").load(str(tmp_path)).writeStream.foreachBatch(func)
+    query = query.start()
+    try:
+        query.processAllAvailable()
+        assert [row.value for row in session.table(table).collect()] == ["c"]
+    finally:
+        query.stop()
+        session.sql(f"DROP TABLE IF EXISTS {table}").collect()
+
+
 def test_pickled_when_columns_chain_on_the_server_as_pyspark(session, 
tmp_path):
     # As PySpark's: a pickled when() Column keeps its CASE WHEN, so it 
collects as the chain
     # built without the pickling, and a foreachBatch function that captures it 
chains on it on
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 440700fd..86dd2d65 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1429,6 +1429,54 @@ def 
test_sql_utils_reexports_the_exceptions_of_pyspark(mock_connect):
     assert "TABLE_OR_VIEW_NOT_FOUND" in str(error.value)
 
 
+def test_pickled_sort_columns_are_sort_orders_as_pyspark(mock_connect):
+    # As PySpark's SortOrder: a pickled, cloudpickled or deep-copied sort 
Column is a sort order
+    # again, so orderBy, Window.orderBy and repartitionByRange send it as the 
Column built
+    # without the pickling, not wrapped in another ASC NULLS FIRST.
+    import copy
+
+    import cloudpickle
+    from pyspark.sql import functions as F
+    from pyspark.sql.window import Window
+
+    spark, service = mock_connect
+    df = spark.range(1)
+
+    def sent_relation(dataframe):
+        # ExecutePlanRequest.plan=3, Plan.root=1.
+        dataframe.collect()
+        return 
parse_message(parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0])
+
+    def sort_orders(column):
+        # Relation.sort=7, Sort.order=2.
+        return parse_message(sent_relation(df.orderBy(column))[7][0])[2]
+
+    def range_partitioning(column):
+        # Relation.repartition_by_expression=27, 
RepartitionByExpression.partition_exprs=2.
+        return parse_message(sent_relation(df.repartitionByRange(2, 
column))[27][0])[2]
+
+    def window_expression(column):
+        # Relation.project=3, Project.expressions=3.
+        over = F.row_number().over(Window.orderBy(column)).alias("r")
+        return parse_message(sent_relation(df.select(over))[3][0])[3]
+
+    for direct in (F.col("id").asc(), F.col("id").desc(), 
F.col("id").asc_nulls_last(),
+                   F.col("id").desc_nulls_first(), F.desc("id")):
+        for copied in (pickle.loads(pickle.dumps(direct)),
+                       cloudpickle.loads(cloudpickle.dumps(direct)),
+                       copy.deepcopy(direct)):
+            assert repr(copied) == repr(direct)
+            assert repr(Window.orderBy(copied)) == repr(Window.orderBy(direct))
+            assert sort_orders(copied) == sort_orders(direct)
+            assert range_partitioning(copied) == range_partitioning(direct)
+            assert window_expression(copied) == window_expression(direct)
+    # A sort order of an expression, and one pickled twice.
+    direct = (F.col("id") + 1).desc()
+    twice = pickle.loads(pickle.dumps(pickle.loads(pickle.dumps(direct))))
+    assert repr(twice) == repr(direct)
+    assert sort_orders(twice) == sort_orders(direct)
+
+
 def test_pickled_when_columns_chain_as_pyspark(mock_connect):
     # As PySpark's CaseWhen: a pickled, cloudpickled or deep-copied when() 
Column keeps its CASE
     # WHEN, so when() and otherwise() chain on it, and the chain sends the 
plan of the chain


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to