This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 4d2b073 [SPARK-61121][CONNECT][PYTHON] Take a float frame bound past
its threshold as unbounded in rowsBetween and rangeBetween, in the Python
client as PySpark does
4d2b073 is described below
commit 4d2b073bb866261f7a2d011456300f81174a25fe
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 05:43:36 2026 +0900
[SPARK-61121][CONNECT][PYTHON] Take a float frame bound past its threshold
as unbounded in rowsBetween and rangeBetween, in the Python client as PySpark
does
### What changes were proposed in this pull request?
[SPARK-61121](https://issues.apache.org/jira/browse/SPARK-61121): A float
bound of `rowsBetween` and `rangeBetween` past the threshold of its side is
unbounded, as PySpark's.
PySpark compares the bounds with `Window._PRECEDING_THRESHOLD` and
`_FOLLOWING_THRESHOLD` before `WindowFrame` asserts an int:
- A start at or below the first is unbounded preceding, also for a float.
- An end at or above the second is unbounded following, also for a float.
So `w.rowsBetween(float("-inf"), float("inf"))` is a frame of unbounded
bounds. The client now does the same, and still raises `AssertionError` for
another float, such as `1.5`, a start of `inf` or an end of `-inf`.
### Why are the changes needed?
The client raised `AssertionError` for any float bound, also an infinite
one. Apache Spark's `test_window_functions` and
`test_window_functions_without_partitionBy` use infinite bounds, and failed on
this client.
### Does this PR introduce _any_ user-facing change?
Yes. Infinite and other out-of-range float bounds are unbounded, as with
PySpark 4.3.0.
### How was this patch tested?
- The offline test of window frames checks:
- the frames of infinite and out-of-range float bounds, which are those
of `unboundedPreceding` and `unboundedFollowing`
- the `AssertionError` of other floats
The cases give the same results with official PySpark 4.3.0.
- Apache Spark's `test_parity_functions.py`, run on this client against a
local server, now passes `test_window_functions` and
`test_window_functions_without_partitionBy`.
- The lib tests of both crates (590 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2165 passed, 8 skipped), and the
live drop-in and Arrow suites (173 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #316 from HyukjinKwon/SPARK-61121-window-inf.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/window.rs | 14 +++++++++++---
python/tests/test_dropin_offline.py | 10 ++++++++++
2 files changed, 21 insertions(+), 3 deletions(-)
diff --git a/crates/pyspark-rs/src/window.rs b/crates/pyspark-rs/src/window.rs
index 326f46a..a4fb577 100644
--- a/crates/pyspark-rs/src/window.rs
+++ b/crates/pyspark-rs/src/window.rs
@@ -83,15 +83,23 @@ const UNBOUNDED_FOLLOWING: i64 = i64::MAX;
/// Resolve a frame-bound argument, the `start` of a frame if `start`, else
its end: a plain
/// int (negative=preceding/positive=following) or a `FrameBound`. As PySpark's
/// `rowsBetween` and `rangeBetween`, a start of `-sys.maxsize` or less is
unbounded preceding
-/// and an end of `sys.maxsize` or more unbounded following, while a start of
-/// `Window.unboundedFollowing` or an end of `Window.unboundedPreceding` is a
value.
+/// and an end of `sys.maxsize` or more unbounded following, also of a float,
such as
+/// `float("-inf")` and `float("inf")`, while a start of
`Window.unboundedFollowing` or an end of
+/// `Window.unboundedPreceding` is a value.
fn to_frame_bound(v: &Bound<'_, PyAny>, start: bool) -> PyResult<FrameBound> {
if let Ok(fb) = v.extract::<PyFrameBound>() {
return Ok(fb.bound);
}
if !v.is_instance_of::<pyo3::types::PyInt>() {
- // PySpark's WindowFrame asserts an int, after a comparison that a
number makes.
if v.is_instance_of::<pyo3::types::PyFloat>() {
+ // As PySpark's: a bound past the threshold of its side is
unbounded, and
+ // WindowFrame asserts that another is an int.
+ if start && v.le(-i64::MAX)? {
+ return Ok(FrameBound::UnboundedPreceding);
+ }
+ if !start && v.ge(i64::MAX)? {
+ return Ok(FrameBound::UnboundedFollowing);
+ }
return Err(pyo3::exceptions::PyAssertionError::new_err(()));
}
return Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 9a7f7c7..b694a16 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -9007,6 +9007,16 @@ def
test_window_frames_and_columns_are_checked_as_pyspark():
"arg_name": name, "min": "-2147483648", "max": "2147483647"}
with pytest.raises(AssertionError):
o.rowsBetween(1.5, 2)
+ # A float bound past the threshold of its side is unbounded, as PySpark
compares it with
+ # Window._PRECEDING_THRESHOLD and _FOLLOWING_THRESHOLD before WindowFrame
asserts an int.
+ inf = float("inf")
+ unbounded = (Window.unboundedPreceding, Window.unboundedFollowing)
+ assert repr(o.rowsBetween(-inf, inf)) == repr(o.rowsBetween(*unbounded))
+ assert repr(o.rangeBetween(-inf, inf)) == repr(o.rangeBetween(*unbounded))
+ assert repr(o.rowsBetween(-1e30, 1e30)) == repr(o.rowsBetween(*unbounded))
+ for start, end in ((inf, 0), (0, -inf), (-1e30, 1.5)):
+ with pytest.raises(AssertionError):
+ o.rowsBetween(start, end)
for call, arg_type in ((lambda: Window.partitionBy(1), "int"),
(lambda: Window.partitionBy(("id",)), "tuple"),
(lambda: Window.orderBy(1), "int"),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]