This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 4d2b073  [SPARK-61121][CONNECT][PYTHON] Take a float frame bound past 
its threshold as unbounded in rowsBetween and rangeBetween, in the Python 
client as PySpark does
4d2b073 is described below

commit 4d2b073bb866261f7a2d011456300f81174a25fe
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 05:43:36 2026 +0900

    [SPARK-61121][CONNECT][PYTHON] Take a float frame bound past its threshold 
as unbounded in rowsBetween and rangeBetween, in the Python client as PySpark 
does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61121](https://issues.apache.org/jira/browse/SPARK-61121): A float 
bound of `rowsBetween` and `rangeBetween` past the threshold of its side is 
unbounded, as PySpark's.
    
    PySpark compares the bounds with `Window._PRECEDING_THRESHOLD` and 
`_FOLLOWING_THRESHOLD` before `WindowFrame` asserts an int:
    - A start at or below the first is unbounded preceding, also for a float.
    - An end at or above the second is unbounded following, also for a float.
    
    So `w.rowsBetween(float("-inf"), float("inf"))` is a frame of unbounded 
bounds. The client now does the same, and still raises `AssertionError` for 
another float, such as `1.5`, a start of `inf` or an end of `-inf`.
    
    ### Why are the changes needed?
    
    The client raised `AssertionError` for any float bound, also an infinite 
one. Apache Spark's `test_window_functions` and 
`test_window_functions_without_partitionBy` use infinite bounds, and failed on 
this client.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Infinite and other out-of-range float bounds are unbounded, as with 
PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - The offline test of window frames checks:
      - the frames of infinite and out-of-range float bounds, which are those 
of `unboundedPreceding` and `unboundedFollowing`
      - the `AssertionError` of other floats
    
      The cases give the same results with official PySpark 4.3.0.
    - Apache Spark's `test_parity_functions.py`, run on this client against a 
local server, now passes `test_window_functions` and 
`test_window_functions_without_partitionBy`.
    - The lib tests of both crates (590 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2165 passed, 8 skipped), and the 
live drop-in and Arrow suites (173 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #316 from HyukjinKwon/SPARK-61121-window-inf.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/window.rs     | 14 +++++++++++---
 python/tests/test_dropin_offline.py | 10 ++++++++++
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/crates/pyspark-rs/src/window.rs b/crates/pyspark-rs/src/window.rs
index 326f46a..a4fb577 100644
--- a/crates/pyspark-rs/src/window.rs
+++ b/crates/pyspark-rs/src/window.rs
@@ -83,15 +83,23 @@ const UNBOUNDED_FOLLOWING: i64 = i64::MAX;
 /// Resolve a frame-bound argument, the `start` of a frame if `start`, else 
its end: a plain
 /// int (negative=preceding/positive=following) or a `FrameBound`. As PySpark's
 /// `rowsBetween` and `rangeBetween`, a start of `-sys.maxsize` or less is 
unbounded preceding
-/// and an end of `sys.maxsize` or more unbounded following, while a start of
-/// `Window.unboundedFollowing` or an end of `Window.unboundedPreceding` is a 
value.
+/// and an end of `sys.maxsize` or more unbounded following, also of a float, 
such as
+/// `float("-inf")` and `float("inf")`, while a start of 
`Window.unboundedFollowing` or an end of
+/// `Window.unboundedPreceding` is a value.
 fn to_frame_bound(v: &Bound<'_, PyAny>, start: bool) -> PyResult<FrameBound> {
     if let Ok(fb) = v.extract::<PyFrameBound>() {
         return Ok(fb.bound);
     }
     if !v.is_instance_of::<pyo3::types::PyInt>() {
-        // PySpark's WindowFrame asserts an int, after a comparison that a 
number makes.
         if v.is_instance_of::<pyo3::types::PyFloat>() {
+            // As PySpark's: a bound past the threshold of its side is 
unbounded, and
+            // WindowFrame asserts that another is an int.
+            if start && v.le(-i64::MAX)? {
+                return Ok(FrameBound::UnboundedPreceding);
+            }
+            if !start && v.ge(i64::MAX)? {
+                return Ok(FrameBound::UnboundedFollowing);
+            }
             return Err(pyo3::exceptions::PyAssertionError::new_err(()));
         }
         return Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 9a7f7c7..b694a16 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9007,6 +9007,16 @@ def 
test_window_frames_and_columns_are_checked_as_pyspark():
             "arg_name": name, "min": "-2147483648", "max": "2147483647"}
     with pytest.raises(AssertionError):
         o.rowsBetween(1.5, 2)
+    # A float bound past the threshold of its side is unbounded, as PySpark 
compares it with
+    # Window._PRECEDING_THRESHOLD and _FOLLOWING_THRESHOLD before WindowFrame 
asserts an int.
+    inf = float("inf")
+    unbounded = (Window.unboundedPreceding, Window.unboundedFollowing)
+    assert repr(o.rowsBetween(-inf, inf)) == repr(o.rowsBetween(*unbounded))
+    assert repr(o.rangeBetween(-inf, inf)) == repr(o.rangeBetween(*unbounded))
+    assert repr(o.rowsBetween(-1e30, 1e30)) == repr(o.rowsBetween(*unbounded))
+    for start, end in ((inf, 0), (0, -inf), (-1e30, 1.5)):
+        with pytest.raises(AssertionError):
+            o.rowsBetween(start, end)
     for call, arg_type in ((lambda: Window.partitionBy(1), "int"),
                            (lambda: Window.partitionBy(("id",)), "tuple"),
                            (lambda: Window.orderBy(1), "int"),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to