This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 4bbfaffd [SPARK-61191][CONNECT][PYTHON] Pickle WindowSpec, so UDFs and
foreachBatch functions can capture windows, in the Python client as PySpark does
4bbfaffd is described below
commit 4bbfaffd919e7aa798fdc9cbafd094973a829fae
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 19:52:49 2026 +0900
[SPARK-61191][CONNECT][PYTHON] Pickle WindowSpec, so UDFs and foreachBatch
functions can capture windows, in the Python client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61191](https://issues.apache.org/jira/browse/SPARK-61191): Pickle
`WindowSpec`, so UDFs and `foreachBatch` functions can capture windows, as
PySpark does.
`PyWindowSpec` gets a `__reduce__`. It pickles the partition Columns, the
sort orders and the frame of the spec, and
`pyspark.sql.connect._window_reduce.window_of` gives back an equal spec through
`Window`:
- a sort order is pickled as the Column of its child and the method of its
direction and nulls (`desc_nulls_last`, ...), since an unpickled Column does
not keep its sort order, and the method rebuilds it on the client and on the
worker alike;
- the frame is pickled as its ROWS flag and its start and end as the ints
of `Window`, so the unbounded and current-row bounds, and `rowsBetween` vs
`rangeBetween`, are kept.
The `window_of` function is registered with cloudpickle to pickle by value,
as the Column reducer is, so it runs on a Python worker of the server, where
PySpark's `Window` builds the spec. Official PySpark's WindowSpec is a plain
Python object that pickles the same way.
### Why are the changes needed?
`pickle.dumps(Window.partitionBy("k").orderBy("x"))`,
`cloudpickle.dumps(...)` and `copy.copy(...)` raised `TypeError: cannot pickle
'pyspark.sql.window.WindowSpec' object`. A `foreachBatch` or UDF function that
captures a window, as `cloudpickle.dumps(lambda df:
df.select(F.row_number().over(w)))`, therefore failed at serialization, where
official PySpark runs it.
### Does this PR introduce _any_ user-facing change?
Yes. `WindowSpec` pickles, deep-copies and shallow-copies, with the same
repr and the same window in the plan, and UDFs and `foreachBatch` functions can
capture window specs.
### How was this patch tested?
- `test_window_specs_pickle_as_pyspark` (offline, mock server): for
partition, order (`desc`, `asc_nulls_last`, `desc_nulls_first`, plain names),
ROWS and RANGE frames with unbounded and current-row bounds, and an empty spec,
pickle, cloudpickle and copy round trips have the same repr and send the same
`F.sum(...).over(...)` expression as the spec. Fails without the fix.
- `test_window_specs_captured_by_foreach_batch_functions_run_as_pyspark`
(live): a `foreachBatch` function that captures a window and computes
`row_number().over(window)` on the server's worker writes the expected rows.
Fails without the fix.
- The same specs give the same reprs and the same results (the `over` of
each, and a cloudpickled closure over a window) with official PySpark 4.3.0.
- The lib tests of both crates (599 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2228 passed, 8 skipped), and
the live tests of windows, foreachBatch, pickling and UDFs (16 passed) against
a local Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics
in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #371 from HyukjinKwon/SPARK-61191-pickle-window-spec.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/window.rs | 58 +++++++++++++++++++++++++++-
python/pyspark/sql/connect/_window_reduce.py | 45 +++++++++++++++++++++
python/pyspark/sql/connect/window.py | 4 ++
python/tests/test_dropin_e2e.py | 29 ++++++++++++++
python/tests/test_dropin_offline.py | 36 +++++++++++++++++
5 files changed, 170 insertions(+), 2 deletions(-)
diff --git a/crates/pyspark-rs/src/window.rs b/crates/pyspark-rs/src/window.rs
index fff2d9ba..5d5239a0 100644
--- a/crates/pyspark-rs/src/window.rs
+++ b/crates/pyspark-rs/src/window.rs
@@ -1,8 +1,12 @@
//! PyO3 wrappers for window functions and specifications.
use pyo3::prelude::*;
-use spark_connect::expression::{Expression, ExpressionNode, SortOrder};
-use spark_connect::window::{FrameBound, WindowSpec};
+use pyo3::types::PyTuple;
+use spark_connect::column::Column;
+use spark_connect::expression::{Expression, ExpressionNode, NullOrdering,
SortOrder};
+use spark_connect::window::{FrameBound, FrameType, WindowSpec};
+
+use crate::column::PyColumn;
/// The columns of `Window.partitionBy` and `orderBy`, as PySpark Connect's
`_to_cols`: a single
/// list is the columns, each a Column or a column name (not a literal), else
@@ -229,6 +233,56 @@ impl PyWindowSpec {
fn __repr__(&self) -> String {
self.spec.repr()
}
+
+ /// A WindowSpec pickles as the partition and sort Columns and the frame
of it, which the
+ /// function `pyspark.sql.connect._window_reduce.window_of` gives back as
a WindowSpec built
+ /// through `Window`, also on a Python worker of the server, as PySpark's
WindowSpec pickles
+ /// the Columns it holds. A sort order is given as the Column of its child
and the method of
+ /// its direction and nulls, as an unpickled Column is not a sort order of
its own.
+ fn __reduce__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+ let column = |expr: &Expression| Py::new(py,
PyColumn::new(Column::new(expr.clone())));
+ let partition = self
+ .spec
+ .partition_spec
+ .iter()
+ .map(column)
+ .collect::<PyResult<Vec<_>>>()?;
+ let order = self
+ .spec
+ .order_spec
+ .iter()
+ .map(|sort| {
+ let method = match (sort.ascending, sort.null_ordering) {
+ (true, NullOrdering::First) => "asc_nulls_first",
+ (true, NullOrdering::Last) => "asc_nulls_last",
+ (false, NullOrdering::First) => "desc_nulls_first",
+ (false, NullOrdering::Last) => "desc_nulls_last",
+ };
+ Ok((column(&sort.child)?, method))
+ })
+ .collect::<PyResult<Vec<_>>>()?;
+ // The frame as its ROWS flag and its start and end, as the ints of
`Window`.
+ let bound = |bound: &FrameBound| match bound {
+ FrameBound::UnboundedPreceding => UNBOUNDED_PRECEDING,
+ FrameBound::Preceding(n) => -n,
+ FrameBound::CurrentRow => 0,
+ FrameBound::Following(n) => *n,
+ FrameBound::UnboundedFollowing => UNBOUNDED_FOLLOWING,
+ };
+ let frame = self
+ .spec
+ .frame_spec
+ .as_ref()
+ .map(|(frame_type, start, end)| {
+ (*frame_type == FrameType::Row, bound(start), bound(end))
+ });
+ let window_of = py
+ .import("pyspark.sql.connect.window")?
+ .getattr("_window_reduce")?
+ .getattr("window_of")?;
+ let args = (partition, order, frame).into_pyobject(py)?;
+ Ok(PyTuple::new(py, [window_of, args.into_any()])?.into_any())
+ }
}
/// Python wrapper for the Window static builder.
diff --git a/python/pyspark/sql/connect/_window_reduce.py
b/python/pyspark/sql/connect/_window_reduce.py
new file mode 100644
index 00000000..24a803d9
--- /dev/null
+++ b/python/pyspark/sql/connect/_window_reduce.py
@@ -0,0 +1,45 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""The function that a pickled WindowSpec of the client unpickles with.
+
+``pyspark.sql.connect.window`` registers this module with cloudpickle to
pickle by value, so a
+window that a UDF or a ``foreachBatch`` function captures unpickles on a
Python worker of the
+server, whose PySpark has no Rust client, while ``pickle`` pickles the
function by reference, of
+this module, on the client.
+"""
+from typing import Any, Optional, Sequence, Tuple
+
+
+def window_of(
+ partition: Sequence[Any],
+ order: Sequence[Tuple[Any, str]],
+ frame: Optional[Tuple[bool, int, int]],
+) -> Any:
+ """The WindowSpec of the Columns ``partition`` and of the sort orders
``order``, each a
+ Column of its child and the name of the method of its direction and nulls,
such as
+ ``desc_nulls_last``, and of ``frame``, the ROWS flag and the start and end
of its frame, if
+ any, built through ``Window`` as PySpark's WindowSpec is, on the client
and on a Python worker
+ of the server."""
+ from pyspark.sql.window import Window
+
+ spec = Window.partitionBy(*partition).orderBy(
+ *[getattr(child, method)() for child, method in order]
+ )
+ if frame is not None:
+ is_rows, start, end = frame
+ spec = spec.rowsBetween(start, end) if is_rows else
spec.rangeBetween(start, end)
+ return spec
diff --git a/python/pyspark/sql/connect/window.py
b/python/pyspark/sql/connect/window.py
index 5bfb3aa8..b5448f5f 100644
--- a/python/pyspark/sql/connect/window.py
+++ b/python/pyspark/sql/connect/window.py
@@ -15,6 +15,10 @@
# limitations under the License.
#
"""Window / WindowSpec under the connect import path."""
+from pyspark import cloudpickle as _cloudpickle
+from pyspark.sql.connect import _window_reduce
from pyspark.sql.window import Window, WindowSpec
__all__ = ["Window", "WindowSpec"]
+
+_cloudpickle.register_pickle_by_value(_window_reduce)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 16fdf0c6..ff5f5801 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1472,6 +1472,35 @@ def
test_reader_writer_and_catalog_return_the_results_of_pyspark(session, tmp_pa
session.sql("DROP TABLE IF EXISTS e2e_external_eagerly")
+def
test_window_specs_captured_by_foreach_batch_functions_run_as_pyspark(session,
tmp_path):
+ # As PySpark's: a WindowSpec that a foreachBatch function captures
unpickles on the Python
+ # worker of the server, which computes the window with it.
+ from pyspark.sql import Window
+
+ (tmp_path / "input.txt").write_text("hello\nthis\n")
+ window = (
+ Window.partitionBy()
+ .orderBy(F.col("value").desc())
+ .rowsBetween(Window.unboundedPreceding, Window.currentRow)
+ )
+ table = "foreach_batch_captured_window"
+
+ def func(df, batch_id):
+ df.select("value", F.row_number().over(window).alias("r")).write.mode(
+ "append"
+ ).saveAsTable(table)
+
+ query =
session.readStream.format("text").load(str(tmp_path)).writeStream.foreachBatch(func)
+ query = query.start()
+ try:
+ query.processAllAvailable()
+ rows = session.table(table).collect()
+ assert sorted((row.value, row.r) for row in rows) == [("hello", 2),
("this", 1)]
+ finally:
+ query.stop()
+ session.sql(f"DROP TABLE IF EXISTS {table}").collect()
+
+
def test_table_arguments_are_those_of_pyspark_sql_table_arg(session):
# As the PySpark client: DataFrame.asTable() is a TableArg of
pyspark.sql.table_arg, and a
# TableValuedFunctionArgument of pyspark.sql.tvf_argument.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index a4e4221d..cdeca982 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -2100,6 +2100,42 @@ def test_time_values_are_sent_as_pyspark(mock_connect):
assert parse_message(parse_message(expression[1][0])[26][0]) == {1:
[nanos], 2: [6]}
+def test_window_specs_pickle_as_pyspark(mock_connect):
+ # As PySpark's WindowSpec, which pickles its partition, order and frame: a
pickled, cloudpickled
+ # or copied WindowSpec has the repr of the spec and sends the window of
the spec, so a UDF or
+ # foreachBatch function can capture a window.
+ import copy
+
+ import cloudpickle
+ from pyspark.sql import Window
+ from pyspark.sql import functions as F
+
+ spark, service = mock_connect
+ df = spark.range(1)
+
+ def sent_expression(dataframe):
+ # ExecutePlanRequest.plan=3, Plan.root=1, Relation.project=3,
Project.expressions=3.
+ dataframe.collect()
+ plan =
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+ return parse_message(parse_message(parse_message(plan)[3][0])[3][0])
+
+ specs = [
+ Window.partitionBy("k").orderBy(F.col("x").desc()),
+ Window.partitionBy(F.col("k"), "v").orderBy("x",
F.col("v").asc_nulls_last())
+ .rowsBetween(-1, Window.unboundedFollowing),
+ Window.orderBy("x").rangeBetween(Window.unboundedPreceding, 0),
+
Window.orderBy(F.col("x").desc_nulls_first()).rowsBetween(Window.currentRow, 2),
+ Window.partitionBy(),
+ ]
+ for spec in specs:
+ expected =
sent_expression(df.select(F.sum("id").over(spec).alias("s")))
+ for copied in (pickle.loads(pickle.dumps(spec)),
+ cloudpickle.loads(cloudpickle.dumps(spec)),
+ copy.copy(spec)):
+ assert repr(copied) == repr(spec)
+ assert
sent_expression(df.select(F.sum("id").over(copied).alias("s"))) == expected
+
+
def test_time_types_parse_as_pyspark(mock_connect):
# As PySpark's: a bare time is TimeType(), of the default precision 6, in
a DDL string and
# in the JSON of a type.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]