This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 4bbfaffd [SPARK-61191][CONNECT][PYTHON] Pickle WindowSpec, so UDFs and 
foreachBatch functions can capture windows, in the Python client as PySpark does
4bbfaffd is described below

commit 4bbfaffd919e7aa798fdc9cbafd094973a829fae
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 19:52:49 2026 +0900

    [SPARK-61191][CONNECT][PYTHON] Pickle WindowSpec, so UDFs and foreachBatch 
functions can capture windows, in the Python client as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61191](https://issues.apache.org/jira/browse/SPARK-61191): Pickle 
`WindowSpec`, so UDFs and `foreachBatch` functions can capture windows, as 
PySpark does.
    
    `PyWindowSpec` gets a `__reduce__`. It pickles the partition Columns, the 
sort orders and the frame of the spec, and 
`pyspark.sql.connect._window_reduce.window_of` gives back an equal spec through 
`Window`:
    
    - a sort order is pickled as the Column of its child and the method of its 
direction and nulls (`desc_nulls_last`, ...), since an unpickled Column does 
not keep its sort order, and the method rebuilds it on the client and on the 
worker alike;
    - the frame is pickled as its ROWS flag and its start and end as the ints 
of `Window`, so the unbounded and current-row bounds, and `rowsBetween` vs 
`rangeBetween`, are kept.
    
    The `window_of` function is registered with cloudpickle to pickle by value, 
as the Column reducer is, so it runs on a Python worker of the server, where 
PySpark's `Window` builds the spec. Official PySpark's WindowSpec is a plain 
Python object that pickles the same way.
    
    ### Why are the changes needed?
    
    `pickle.dumps(Window.partitionBy("k").orderBy("x"))`, 
`cloudpickle.dumps(...)` and `copy.copy(...)` raised `TypeError: cannot pickle 
'pyspark.sql.window.WindowSpec' object`. A `foreachBatch` or UDF function that 
captures a window, as `cloudpickle.dumps(lambda df: 
df.select(F.row_number().over(w)))`, therefore failed at serialization, where 
official PySpark runs it.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `WindowSpec` pickles, deep-copies and shallow-copies, with the same 
repr and the same window in the plan, and UDFs and `foreachBatch` functions can 
capture window specs.
    
    ### How was this patch tested?
    
    - `test_window_specs_pickle_as_pyspark` (offline, mock server): for 
partition, order (`desc`, `asc_nulls_last`, `desc_nulls_first`, plain names), 
ROWS and RANGE frames with unbounded and current-row bounds, and an empty spec, 
pickle, cloudpickle and copy round trips have the same repr and send the same 
`F.sum(...).over(...)` expression as the spec. Fails without the fix.
    - `test_window_specs_captured_by_foreach_batch_functions_run_as_pyspark` 
(live): a `foreachBatch` function that captures a window and computes 
`row_number().over(window)` on the server's worker writes the expected rows. 
Fails without the fix.
    - The same specs give the same reprs and the same results (the `over` of 
each, and a cloudpickled closure over a window) with official PySpark 4.3.0.
    - The lib tests of both crates (599 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2228 passed, 8 skipped), and 
the live tests of windows, foreachBatch, pickling and UDFs (16 passed) against 
a local Spark 4.3.0 server; rustfmt passes, and clippy reports no diagnostics 
in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #371 from HyukjinKwon/SPARK-61191-pickle-window-spec.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/window.rs              | 58 +++++++++++++++++++++++++++-
 python/pyspark/sql/connect/_window_reduce.py | 45 +++++++++++++++++++++
 python/pyspark/sql/connect/window.py         |  4 ++
 python/tests/test_dropin_e2e.py              | 29 ++++++++++++++
 python/tests/test_dropin_offline.py          | 36 +++++++++++++++++
 5 files changed, 170 insertions(+), 2 deletions(-)

diff --git a/crates/pyspark-rs/src/window.rs b/crates/pyspark-rs/src/window.rs
index fff2d9ba..5d5239a0 100644
--- a/crates/pyspark-rs/src/window.rs
+++ b/crates/pyspark-rs/src/window.rs
@@ -1,8 +1,12 @@
 //! PyO3 wrappers for window functions and specifications.
 
 use pyo3::prelude::*;
-use spark_connect::expression::{Expression, ExpressionNode, SortOrder};
-use spark_connect::window::{FrameBound, WindowSpec};
+use pyo3::types::PyTuple;
+use spark_connect::column::Column;
+use spark_connect::expression::{Expression, ExpressionNode, NullOrdering, 
SortOrder};
+use spark_connect::window::{FrameBound, FrameType, WindowSpec};
+
+use crate::column::PyColumn;
 
 /// The columns of `Window.partitionBy` and `orderBy`, as PySpark Connect's 
`_to_cols`: a single
 /// list is the columns, each a Column or a column name (not a literal), else
@@ -229,6 +233,56 @@ impl PyWindowSpec {
     fn __repr__(&self) -> String {
         self.spec.repr()
     }
+
+    /// A WindowSpec pickles as the partition and sort Columns and the frame 
of it, which the
+    /// function `pyspark.sql.connect._window_reduce.window_of` gives back as 
a WindowSpec built
+    /// through `Window`, also on a Python worker of the server, as PySpark's 
WindowSpec pickles
+    /// the Columns it holds. A sort order is given as the Column of its child 
and the method of
+    /// its direction and nulls, as an unpickled Column is not a sort order of 
its own.
+    fn __reduce__<'py>(&self, py: Python<'py>) -> PyResult<Bound<'py, PyAny>> {
+        let column = |expr: &Expression| Py::new(py, 
PyColumn::new(Column::new(expr.clone())));
+        let partition = self
+            .spec
+            .partition_spec
+            .iter()
+            .map(column)
+            .collect::<PyResult<Vec<_>>>()?;
+        let order = self
+            .spec
+            .order_spec
+            .iter()
+            .map(|sort| {
+                let method = match (sort.ascending, sort.null_ordering) {
+                    (true, NullOrdering::First) => "asc_nulls_first",
+                    (true, NullOrdering::Last) => "asc_nulls_last",
+                    (false, NullOrdering::First) => "desc_nulls_first",
+                    (false, NullOrdering::Last) => "desc_nulls_last",
+                };
+                Ok((column(&sort.child)?, method))
+            })
+            .collect::<PyResult<Vec<_>>>()?;
+        // The frame as its ROWS flag and its start and end, as the ints of 
`Window`.
+        let bound = |bound: &FrameBound| match bound {
+            FrameBound::UnboundedPreceding => UNBOUNDED_PRECEDING,
+            FrameBound::Preceding(n) => -n,
+            FrameBound::CurrentRow => 0,
+            FrameBound::Following(n) => *n,
+            FrameBound::UnboundedFollowing => UNBOUNDED_FOLLOWING,
+        };
+        let frame = self
+            .spec
+            .frame_spec
+            .as_ref()
+            .map(|(frame_type, start, end)| {
+                (*frame_type == FrameType::Row, bound(start), bound(end))
+            });
+        let window_of = py
+            .import("pyspark.sql.connect.window")?
+            .getattr("_window_reduce")?
+            .getattr("window_of")?;
+        let args = (partition, order, frame).into_pyobject(py)?;
+        Ok(PyTuple::new(py, [window_of, args.into_any()])?.into_any())
+    }
 }
 
 /// Python wrapper for the Window static builder.
diff --git a/python/pyspark/sql/connect/_window_reduce.py 
b/python/pyspark/sql/connect/_window_reduce.py
new file mode 100644
index 00000000..24a803d9
--- /dev/null
+++ b/python/pyspark/sql/connect/_window_reduce.py
@@ -0,0 +1,45 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""The function that a pickled WindowSpec of the client unpickles with.
+
+``pyspark.sql.connect.window`` registers this module with cloudpickle to 
pickle by value, so a
+window that a UDF or a ``foreachBatch`` function captures unpickles on a 
Python worker of the
+server, whose PySpark has no Rust client, while ``pickle`` pickles the 
function by reference, of
+this module, on the client.
+"""
+from typing import Any, Optional, Sequence, Tuple
+
+
+def window_of(
+    partition: Sequence[Any],
+    order: Sequence[Tuple[Any, str]],
+    frame: Optional[Tuple[bool, int, int]],
+) -> Any:
+    """The WindowSpec of the Columns ``partition`` and of the sort orders 
``order``, each a
+    Column of its child and the name of the method of its direction and nulls, 
such as
+    ``desc_nulls_last``, and of ``frame``, the ROWS flag and the start and end 
of its frame, if
+    any, built through ``Window`` as PySpark's WindowSpec is, on the client 
and on a Python worker
+    of the server."""
+    from pyspark.sql.window import Window
+
+    spec = Window.partitionBy(*partition).orderBy(
+        *[getattr(child, method)() for child, method in order]
+    )
+    if frame is not None:
+        is_rows, start, end = frame
+        spec = spec.rowsBetween(start, end) if is_rows else 
spec.rangeBetween(start, end)
+    return spec
diff --git a/python/pyspark/sql/connect/window.py 
b/python/pyspark/sql/connect/window.py
index 5bfb3aa8..b5448f5f 100644
--- a/python/pyspark/sql/connect/window.py
+++ b/python/pyspark/sql/connect/window.py
@@ -15,6 +15,10 @@
 # limitations under the License.
 #
 """Window / WindowSpec under the connect import path."""
+from pyspark import cloudpickle as _cloudpickle
+from pyspark.sql.connect import _window_reduce
 from pyspark.sql.window import Window, WindowSpec
 
 __all__ = ["Window", "WindowSpec"]
+
+_cloudpickle.register_pickle_by_value(_window_reduce)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 16fdf0c6..ff5f5801 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1472,6 +1472,35 @@ def 
test_reader_writer_and_catalog_return_the_results_of_pyspark(session, tmp_pa
         session.sql("DROP TABLE IF EXISTS e2e_external_eagerly")
 
 
+def 
test_window_specs_captured_by_foreach_batch_functions_run_as_pyspark(session, 
tmp_path):
+    # As PySpark's: a WindowSpec that a foreachBatch function captures 
unpickles on the Python
+    # worker of the server, which computes the window with it.
+    from pyspark.sql import Window
+
+    (tmp_path / "input.txt").write_text("hello\nthis\n")
+    window = (
+        Window.partitionBy()
+        .orderBy(F.col("value").desc())
+        .rowsBetween(Window.unboundedPreceding, Window.currentRow)
+    )
+    table = "foreach_batch_captured_window"
+
+    def func(df, batch_id):
+        df.select("value", F.row_number().over(window).alias("r")).write.mode(
+            "append"
+        ).saveAsTable(table)
+
+    query = 
session.readStream.format("text").load(str(tmp_path)).writeStream.foreachBatch(func)
+    query = query.start()
+    try:
+        query.processAllAvailable()
+        rows = session.table(table).collect()
+        assert sorted((row.value, row.r) for row in rows) == [("hello", 2), 
("this", 1)]
+    finally:
+        query.stop()
+        session.sql(f"DROP TABLE IF EXISTS {table}").collect()
+
+
 def test_table_arguments_are_those_of_pyspark_sql_table_arg(session):
     # As the PySpark client: DataFrame.asTable() is a TableArg of 
pyspark.sql.table_arg, and a
     # TableValuedFunctionArgument of pyspark.sql.tvf_argument.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index a4e4221d..cdeca982 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -2100,6 +2100,42 @@ def test_time_values_are_sent_as_pyspark(mock_connect):
     assert parse_message(parse_message(expression[1][0])[26][0]) == {1: 
[nanos], 2: [6]}
 
 
+def test_window_specs_pickle_as_pyspark(mock_connect):
+    # As PySpark's WindowSpec, which pickles its partition, order and frame: a 
pickled, cloudpickled
+    # or copied WindowSpec has the repr of the spec and sends the window of 
the spec, so a UDF or
+    # foreachBatch function can capture a window.
+    import copy
+
+    import cloudpickle
+    from pyspark.sql import Window
+    from pyspark.sql import functions as F
+
+    spark, service = mock_connect
+    df = spark.range(1)
+
+    def sent_expression(dataframe):
+        # ExecutePlanRequest.plan=3, Plan.root=1, Relation.project=3, 
Project.expressions=3.
+        dataframe.collect()
+        plan = 
parse_message(parse_message(service.execute_requests[-1])[3][0])[1][0]
+        return parse_message(parse_message(parse_message(plan)[3][0])[3][0])
+
+    specs = [
+        Window.partitionBy("k").orderBy(F.col("x").desc()),
+        Window.partitionBy(F.col("k"), "v").orderBy("x", 
F.col("v").asc_nulls_last())
+        .rowsBetween(-1, Window.unboundedFollowing),
+        Window.orderBy("x").rangeBetween(Window.unboundedPreceding, 0),
+        
Window.orderBy(F.col("x").desc_nulls_first()).rowsBetween(Window.currentRow, 2),
+        Window.partitionBy(),
+    ]
+    for spec in specs:
+        expected = 
sent_expression(df.select(F.sum("id").over(spec).alias("s")))
+        for copied in (pickle.loads(pickle.dumps(spec)),
+                       cloudpickle.loads(cloudpickle.dumps(spec)),
+                       copy.copy(spec)):
+            assert repr(copied) == repr(spec)
+            assert 
sent_expression(df.select(F.sum("id").over(copied).alias("s"))) == expected
+
+
 def test_time_types_parse_as_pyspark(mock_connect):
     # As PySpark's: a bare time is TimeType(), of the default precision 6, in 
a DDL string and
     # in the JSON of a type.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to