This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 97945273 [SPARK-61199][CONNECT][PYTHON] Keep the sort order of a
pickled sort Column, so that orderBy, Window.orderBy and repartitionByRange
sort by it, in the Python client as PySpark does
97945273 is described below
commit 9794527370c6efc4dd99a4eaaa508764bb9de1b0
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 09:07:52 2026 +0900
[SPARK-61199][CONNECT][PYTHON] Keep the sort order of a pickled sort
Column, so that orderBy, Window.orderBy and repartitionByRange sort by it, in
the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61199](https://issues.apache.org/jira/browse/SPARK-61199): PySpark
Connect's Column pickles its expression, and a sort Column's expression is a
`SortOrder` of its child, its direction and its null ordering. So a pickled
`F.col("x").desc()` is a sort order again wherever it unpickles, and `orderBy`,
`Window.orderBy` and `repartitionByRange` sort by it.
A Column of this client pickles as the bytes of its expression
(SPARK-61167), which unpickle as an opaque expression. Now a sort order pickles
as its child Column and the name of its sort method (`asc_nulls_first`,
`asc_nulls_last`, `desc_nulls_first` or `desc_nulls_last`), as SPARK-61190 did
for CASE WHEN:
- `Column.__reduce__` (Rust) passes `(child, method)` of a sort order to
`_reduce_column`.
- `pyspark.sql.connect._column_reduce.column_of` calls that method on the
unpickled child. On the client the child is a Column of this client. On a
Python worker of the server it is PySpark Connect's Column, whose method gives
back PySpark's `SortOrder`.
### Why are the changes needed?
On a Python worker of the server, PySpark's `DataFrame.orderBy` wraps a
Column that is not a `SortOrder` in `asc()`. So a `foreachBatch` function that
captured a sort Column sent a sort order of a sort order:
```
captured = F.col("value").desc()
def func(batch, batch_id):
batch.orderBy(captured).limit(1).write.mode("append").saveAsTable("t")
# SparkException: [INTERNAL_ERROR] Cannot evaluate expression: ... DESC
NULLS LAST
```
On the client, `Window.orderBy` and `repartitionByRange` wrapped an
unpickled sort Column in ASC NULLS FIRST in the same way:
```
>>> Window.orderBy(pickle.loads(pickle.dumps(F.col("x").desc())))
WindowSpec(OrderBy(x DESC NULLS LAST ASC NULLS FIRST)) # official:
WindowSpec(OrderBy(x DESC NULLS LAST))
>>> df.select(F.row_number().over(that_window)).collect() # a partition
of two or more rows
SparkException: [INTERNAL_ERROR] Cannot evaluate expression: input[1, int,
true] DESC NULLS LAST
```
### Does this PR introduce _any_ user-facing change?
Yes. A pickled, cloudpickled or deep-copied sort Column is a sort order:
`orderBy`, `Window.orderBy` and `repartitionByRange` sort by it on the client,
and so do the DataFrame methods on the Python worker of the server, for a sort
Column that a `foreachBatch` function or a UDF captures.
### How was this patch tested?
- `test_pickled_sort_columns_are_sort_orders_as_pyspark` (offline, mock
server): for `asc`, `desc`, `asc_nulls_last`, `desc_nulls_first` and `F.desc`,
each pickled, cloudpickled and deep-copied, the Column and its `Window.orderBy`
have the reprs of the Column built without the pickling. `orderBy`,
`repartitionByRange` and a window over it also send the same sort orders. A
sort order of an expression, pickled twice, matches too. It fails without the
change, where the `Window.orderBy` o [...]
- `test_pickled_sort_columns_sort_on_the_server_as_pyspark` (live):
`orderBy`, `row_number().over(Window.orderBy(...))` and `repartitionByRange` of
a pickled `desc` and `asc_nulls_last` give the rows of the direct Column. A
`foreachBatch` function that captures `F.col("value").desc()` and sorts by it
on the server writes the largest value. It fails without the change, with the
INTERNAL_ERROR above.
- The same cases with official PySpark 4.3.0 against the same server, for
`desc`, `asc_nulls_last`, `desc_nulls_first` and `asc`: the reprs, `orderBy`,
windows and `repartitionByRange` give identical output.
- The pickling tests of the offline suite (`-k pickl`, 67 passed) and the
pickled `when()` live test.
- The lib tests of both crates (618 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2457 passed, 8 skipped), and the
live drop-in, Arrow and ML suites (215 passed) and live cases (11 passed)
against a local Spark 4.3.0 server. Three live foreachBatch and
progress-handler tests failed with `Python worker failed to connect back` while
the machine was overloaded, and passed when run again. Clippy reports no
diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #388 from HyukjinKwon/SPARK-61199-pickled-sort-orders.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/column.rs | 19 +++++++++--
python/pyspark/sql/connect/_column_reduce.py | 10 +++++-
python/pyspark/sql/connect/column.py | 8 +++--
python/tests/test_dropin_e2e.py | 37 +++++++++++++++++++++
python/tests/test_dropin_offline.py | 48 ++++++++++++++++++++++++++++
5 files changed, 116 insertions(+), 6 deletions(-)
diff --git a/crates/pyspark-rs/src/column.rs b/crates/pyspark-rs/src/column.rs
index bf563d52..08306e63 100644
--- a/crates/pyspark-rs/src/column.rs
+++ b/crates/pyspark-rs/src/column.rs
@@ -3,7 +3,7 @@
use pyo3::prelude::*;
use pyo3::types::PyDict;
use spark_connect::column::Column;
-use spark_connect::expression::{Alias, ExpressionNode};
+use spark_connect::expression::{Alias, ExpressionNode, NullOrdering};
use crate::functions::to_column;
@@ -692,7 +692,9 @@ impl PyColumn {
/// `pyspark.sql.connect.column._reduce_column` gives back the Column,
also on a Python worker
/// of the server, such as that of a `foreachBatch` function, as PySpark
Connect's Column.
/// A CASE WHEN also pickles its branches and else value, as PySpark's
CaseWhen keeps them, so
- /// that `when` and `otherwise` chain on the unpickled Column.
+ /// that `when` and `otherwise` chain on the unpickled Column. A sort
order pickles its child
+ /// Column and its sort method, as PySpark's SortOrder keeps its child and
direction, so that
+ /// the unpickled Column is a sort order of `orderBy`, `Window.orderBy`
and the others.
fn __reduce__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
use prost::Message;
let expression = pyo3::types::PyBytes::new(py,
&self.column.to_proto().encode_to_vec());
@@ -715,6 +717,18 @@ impl PyColumn {
}
_ => (None, None),
};
+ let sort = match self.column.expression().node() {
+ ExpressionNode::SortOrder(sort) => {
+ let method = match (sort.ascending, sort.null_ordering) {
+ (true, NullOrdering::First) => "asc_nulls_first",
+ (true, NullOrdering::Last) => "asc_nulls_last",
+ (false, NullOrdering::First) => "desc_nulls_first",
+ (false, NullOrdering::Last) => "desc_nulls_last",
+ };
+ Some((column(&sort.child)?, method))
+ }
+ _ => None,
+ };
py.import("pyspark.sql.connect.column")?
.getattr("_reduce_column")?
.call1((
@@ -722,6 +736,7 @@ impl PyColumn {
self.column.expression().render(),
branches,
else_value,
+ sort,
))
}
diff --git a/python/pyspark/sql/connect/_column_reduce.py
b/python/pyspark/sql/connect/_column_reduce.py
index ff047da2..d9a6cfbe 100644
--- a/python/pyspark/sql/connect/_column_reduce.py
+++ b/python/pyspark/sql/connect/_column_reduce.py
@@ -29,13 +29,21 @@ def column_of(
rendered: str,
branches: Optional[List[Tuple[Any, Any]]] = None,
else_value: Any = None,
+ sort: Optional[Tuple[Any, str]] = None,
) -> Any:
"""The Column of ``expression``, the bytes of a
``spark.connect.Expression``, and of its
rendering ``rendered``, which sends the expression as it is: of the Rust
client, or, on a
Python worker of the server, PySpark Connect's Column of the expression,
as PySpark Connect's
Column of its expression unpickles there, so everything that it calls
there is of PySpark.
For a CASE WHEN, ``branches`` are the (condition, value) Columns of its
branches and
- ``else_value`` the Column of its else value, so that it is a CASE WHEN
again."""
+ ``else_value`` the Column of its else value, so that it is a CASE WHEN
again. For a sort
+ order, ``sort`` is its child Column and the name of its sort method, such
as
+ ``desc_nulls_last``, so that it is a sort order again."""
+ if sort is not None:
+ # The child is a Column of the client, or PySpark Connect's on a
worker, whose sort
+ # method gives back the sort order, as PySpark's SortOrder of its
child.
+ child, method = sort
+ return getattr(child, method)()
try:
from pyspark import _pyspark
except ImportError:
diff --git a/python/pyspark/sql/connect/column.py
b/python/pyspark/sql/connect/column.py
index d40afba0..8d4759e0 100644
--- a/python/pyspark/sql/connect/column.py
+++ b/python/pyspark/sql/connect/column.py
@@ -31,12 +31,14 @@ def _reduce_column(
rendered: str,
branches: Optional[List[Tuple[Any, Any]]] = None,
else_value: Any = None,
+ sort: Optional[Tuple[Any, str]] = None,
) -> Tuple[Callable[..., Any], Tuple[Any, ...]]:
"""The ``__reduce__`` of a Column of the bytes of its
``spark.connect.Expression`` and of its
rendering, as PySpark Connect's Column pickles its expression: the
function of
``pyspark.sql.connect._column_reduce``, which gives back the Column, which
sends the
expression as it is. A CASE WHEN gives its branches and else value too, as
Columns, so the
- Column gives back a CASE WHEN. cloudpickle pickles that function by value,
also to a Python
- worker of the server, where it gives back PySpark Connect's Column, and
``pickle`` by
+ Column gives back a CASE WHEN. A sort order gives its child Column and its
sort method, so
+ the Column gives back a sort order. cloudpickle pickles that function by
value, also to a
+ Python worker of the server, where it gives back PySpark Connect's Column,
and ``pickle`` by
reference."""
- return _column_reduce.column_of, (expression, rendered, branches,
else_value)
+ return _column_reduce.column_of, (expression, rendered, branches,
else_value, sort)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index efddc6bc..7bc7a6b7 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -935,6 +935,43 @@ ARROW_TABLE_ROWS = [
]
+def test_pickled_sort_columns_sort_on_the_server_as_pyspark(session, tmp_path):
+ # As PySpark's: a pickled sort Column is a sort order again, so orderBy, a
window and
+ # repartitionByRange sort by it as by the Column built without the
pickling, and a
+ # foreachBatch function that captures it sorts by it on the Python worker
of the server.
+ import pickle
+
+ from pyspark.sql.window import Window
+
+ df = session.createDataFrame([(1, "a"), (3, "b"), (None, "c"), (2, "d")],
"x int, y string")
+ for direct in (F.col("x").desc(), F.col("x").asc_nulls_last()):
+ copied = pickle.loads(pickle.dumps(direct))
+ assert df.orderBy(copied).collect() == df.orderBy(direct).collect()
+ ranked = df.select("y",
F.row_number().over(Window.orderBy(copied)).alias("r"))
+ assert sorted(ranked.collect()) == sorted(
+ df.select("y",
F.row_number().over(Window.orderBy(direct)).alias("r")).collect()
+ )
+ by_y = lambda row: row.y # noqa: E731
+ assert sorted(df.repartitionByRange(2, copied).collect(), key=by_y) ==
sorted(
+ df.collect(), key=by_y)
+
+ (tmp_path / "input.txt").write_text("a\nc\nb\n")
+ captured = F.col("value").desc()
+ table = "foreach_batch_captured_sort"
+
+ def func(batch, batch_id):
+
batch.orderBy(captured).limit(1).write.mode("append").saveAsTable(table)
+
+ query =
session.readStream.format("text").load(str(tmp_path)).writeStream.foreachBatch(func)
+ query = query.start()
+ try:
+ query.processAllAvailable()
+ assert [row.value for row in session.table(table).collect()] == ["c"]
+ finally:
+ query.stop()
+ session.sql(f"DROP TABLE IF EXISTS {table}").collect()
+
+
def test_pickled_when_columns_chain_on_the_server_as_pyspark(session,
tmp_path):
# As PySpark's: a pickled when() Column keeps its CASE WHEN, so it
collects as the chain
# built without the pickling, and a foreachBatch function that captures it
chains on it on
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 440700fd..86dd2d65 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1429,6 +1429,54 @@ def
test_sql_utils_reexports_the_exceptions_of_pyspark(mock_connect):
assert "TABLE_OR_VIEW_NOT_FOUND" in str(error.value)
+def test_pickled_sort_columns_are_sort_orders_as_pyspark(mock_connect):
+ # As PySpark's SortOrder: a pickled, cloudpickled or deep-copied sort
Column is a sort order
+ # again, so orderBy, Window.orderBy and repartitionByRange send it as the
Column built
+ # without the pickling, not wrapped in another ASC NULLS FIRST.
+ import copy
+
+ import cloudpickle
+ from pyspark.sql import functions as F
+ from pyspark.sql.window import Window
+
+ spark, service = mock_connect
+ df = spark.range(1)
+
+ def sent_relation(dataframe):
+ # ExecutePlanRequest.plan=3, Plan.root=1.
+ dataframe.collect()
+ return
parse_message(parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0])
+
+ def sort_orders(column):
+ # Relation.sort=7, Sort.order=2.
+ return parse_message(sent_relation(df.orderBy(column))[7][0])[2]
+
+ def range_partitioning(column):
+ # Relation.repartition_by_expression=27,
RepartitionByExpression.partition_exprs=2.
+ return parse_message(sent_relation(df.repartitionByRange(2,
column))[27][0])[2]
+
+ def window_expression(column):
+ # Relation.project=3, Project.expressions=3.
+ over = F.row_number().over(Window.orderBy(column)).alias("r")
+ return parse_message(sent_relation(df.select(over))[3][0])[3]
+
+ for direct in (F.col("id").asc(), F.col("id").desc(),
F.col("id").asc_nulls_last(),
+ F.col("id").desc_nulls_first(), F.desc("id")):
+ for copied in (pickle.loads(pickle.dumps(direct)),
+ cloudpickle.loads(cloudpickle.dumps(direct)),
+ copy.deepcopy(direct)):
+ assert repr(copied) == repr(direct)
+ assert repr(Window.orderBy(copied)) == repr(Window.orderBy(direct))
+ assert sort_orders(copied) == sort_orders(direct)
+ assert range_partitioning(copied) == range_partitioning(direct)
+ assert window_expression(copied) == window_expression(direct)
+ # A sort order of an expression, and one pickled twice.
+ direct = (F.col("id") + 1).desc()
+ twice = pickle.loads(pickle.dumps(pickle.loads(pickle.dumps(direct))))
+ assert repr(twice) == repr(direct)
+ assert sort_orders(twice) == sort_orders(direct)
+
+
def test_pickled_when_columns_chain_as_pyspark(mock_connect):
# As PySpark's CaseWhen: a pickled, cloudpickled or deep-copied when()
Column keeps its CASE
# WHEN, so when() and otherwise() chain on it, and the chain sends the
plan of the chain
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]