This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 0cffbce [SPARK-61044][CONNECT][PYTHON] Send the frame boundaries of
rowsBetween and rangeBetween, and check the columns of Window, in the Python
client as PySpark does
0cffbce is described below
commit 0cffbce297ed2f5ac3a90ed424beb524026532f0
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 11:25:49 2026 +0900
[SPARK-61044][CONNECT][PYTHON] Send the frame boundaries of rowsBetween and
rangeBetween, and check the columns of Window, in the Python client as PySpark
does
### What changes were proposed in this pull request?
[SPARK-61044](https://issues.apache.org/jira/browse/SPARK-61044):
`rowsBetween` and `rangeBetween` now send their frame boundaries, and `Window`
checks its columns, as PySpark Connect does.
- A range frame sends the boundaries of its values as long literals, and a
row frame as int literals, as PySpark's `WindowExpression` does. A boundary is
no longer truncated to an int.
- As PySpark's `rowsBetween` and `rangeBetween`, a `start` of
`-sys.maxsize` or less is unbounded preceding, and an `end` of `sys.maxsize` or
more unbounded following. A `start` of `Window.unboundedFollowing`, or an `end`
of `Window.unboundedPreceding`, is a value.
- A row frame raises `VALUE_NOT_BETWEEN` for a boundary out of the range of
an int, and a frame of a float asserts an int. The client raises these when the
frame is made; PySpark raises them when it sends the plan.
- `Window.partitionBy` and `orderBy` raise `NOT_EXPECTED_TYPE` of the
argument `col`, as PySpark's `F._to_col` does, for a column that is neither a
Column nor a column name, such as a tuple.
### Why are the changes needed?
The client sent each boundary as an int literal of its value truncated to
an int. So `Window.orderBy("id").rangeBetween(-(1 << 40), 0)` was the frame of
the current row only: `sum("id")` over it was `[0, 1, 2, 3]` rather than `[0,
1, 3, 6]`. These wrong results are the main fix.
It also:
- raised `TypeError`s for `rowsBetween(-sys.maxsize, sys.maxsize)` and for
boundaries out of the range of an int
- took `Window.unboundedFollowing` as a start, and
`Window.unboundedPreceding` as an end, as unbounded
- took a tuple of columns, and named the argument of its errors `cols`
### Does this PR introduce _any_ user-facing change?
Yes. Window frames return the rows of PySpark 4.3.0, and these methods
raise its errors.
### How was this patch tested?
- A new live test checks the sums over range frames of boundaries out of
the range of an int, and over unbounded frames, and the `VALUE_NOT_BETWEEN` of
a row frame. It passes with official PySpark 4.3.0, and fails without this
change.
- A new offline test checks the frames and their errors, and the errors of
the columns.
- A new lib test checks the literals of the boundaries of row and range
frames.
- On a local server, 19 cases of window frames and columns behave as in
official PySpark 4.3.0, and 5 frames have its reprs.
- The lib tests of both crates (572 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (1811 passed, 8 skipped), and the
live drop-in and Arrow suites (109 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #244 from HyukjinKwon/SPARK-61044-window-frames.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/window.rs | 127 +++++++++++++++++++++++++++------
crates/spark-connect/src/expression.rs | 45 +++++++-----
crates/spark-connect/src/window.rs | 46 ++++++++++++
python/tests/test_dropin_e2e.py | 22 ++++++
python/tests/test_dropin_offline.py | 39 ++++++++++
5 files changed, 240 insertions(+), 39 deletions(-)
diff --git a/crates/pyspark-rs/src/window.rs b/crates/pyspark-rs/src/window.rs
index 9d96729..326f46a 100644
--- a/crates/pyspark-rs/src/window.rs
+++ b/crates/pyspark-rs/src/window.rs
@@ -4,12 +4,45 @@ use pyo3::prelude::*;
use spark_connect::expression::{Expression, ExpressionNode, SortOrder};
use spark_connect::window::{FrameBound, WindowSpec};
-use crate::dataframe::to_column_list;
+/// The columns of `Window.partitionBy` and `orderBy`, as PySpark Connect's
`_to_cols`: a single
+/// list is the columns, each a Column or a column name (not a literal), else
+/// `NOT_EXPECTED_TYPE` of the argument `col`, as its `F._to_col`.
+fn window_columns(cols: Vec<Bound<'_, PyAny>>) ->
PyResult<Vec<spark_connect::column::Column>> {
+ let cols = match cols.as_slice() {
+ [only] if only.is_instance_of::<pyo3::types::PyList>() => {
+ only.try_iter()?.collect::<PyResult<Vec<_>>>()?
+ }
+ _ => cols,
+ };
+ cols.iter()
+ .map(|c| {
+ if let Ok(column) = c.extract::<PyRef<'_,
crate::column::PyColumn>>() {
+ return Ok(column.column.clone());
+ }
+ if c.is_instance_of::<pyo3::types::PyString>() {
+ // As PySpark's `F._to_col`, by `col`, of its origin.
+ let _origin = crate::origin::with_origin("col");
+ return
Ok(spark_connect::functions::col(&c.extract::<String>()?));
+ }
+ let type_name = c.get_type().name()?.to_string();
+ Err(crate::column::raise_pyspark(
+ c.py(),
+ "PySparkTypeError",
+ "NOT_EXPECTED_TYPE",
+ &[
+ ("expected_type", "Column or str"),
+ ("arg_name", "col"),
+ ("arg_type", &type_name),
+ ],
+ ))
+ })
+ .collect()
+}
/// Build partition expressions from ColumnOrName args (a bare string is a
COLUMN
/// NAME, not a literal — mirrors pyspark Window.partitionBy).
fn to_partition_exprs(cols: Vec<Bound<'_, PyAny>>) ->
PyResult<Vec<Expression>> {
- Ok(to_column_list(cols)?
+ Ok(window_columns(cols)?
.iter()
.map(|c| c.expression().clone())
.collect())
@@ -20,7 +53,7 @@ fn to_partition_exprs(cols: Vec<Bound<'_, PyAny>>) ->
PyResult<Vec<Expression>>
/// ascending with nulls first, as `SortOrder(expr)` of PySpark defaults to.
fn to_sort_orders(cols: Vec<Bound<'_, PyAny>>) -> PyResult<Vec<SortOrder>> {
let mut sort_orders = Vec::new();
- for column in to_column_list(cols)? {
+ for column in window_columns(cols)? {
if let ExpressionNode::SortOrder(sort) = column.expression().node() {
sort_orders.push((**sort).clone());
} else {
@@ -47,24 +80,70 @@ impl PyFrameBound {
const UNBOUNDED_PRECEDING: i64 = i64::MIN;
const UNBOUNDED_FOLLOWING: i64 = i64::MAX;
-/// Resolve a frame-bound argument that is either a plain int (pyspark style,
with
-/// sentinel extremes and negative=preceding/positive=following) or a
`FrameBound`.
-fn to_frame_bound(v: &Bound<'_, PyAny>) -> PyResult<FrameBound> {
+/// Resolve a frame-bound argument, the `start` of a frame if `start`, else
its end: a plain
+/// int (negative=preceding/positive=following) or a `FrameBound`. As PySpark's
+/// `rowsBetween` and `rangeBetween`, a start of `-sys.maxsize` or less is
unbounded preceding
+/// and an end of `sys.maxsize` or more unbounded following, while a start of
+/// `Window.unboundedFollowing` or an end of `Window.unboundedPreceding` is a
value.
+fn to_frame_bound(v: &Bound<'_, PyAny>, start: bool) -> PyResult<FrameBound> {
if let Ok(fb) = v.extract::<PyFrameBound>() {
return Ok(fb.bound);
}
- let n: i64 = v.extract().map_err(|_| {
- PyErr::new::<pyo3::exceptions::PyTypeError, _>("frame bound must be an
int or FrameBound")
- })?;
+ if !v.is_instance_of::<pyo3::types::PyInt>() {
+ // PySpark's WindowFrame asserts an int, after a comparison that a
number makes.
+ if v.is_instance_of::<pyo3::types::PyFloat>() {
+ return Err(pyo3::exceptions::PyAssertionError::new_err(()));
+ }
+ return Err(PyErr::new::<pyo3::exceptions::PyTypeError, _>(
+ "frame bound must be an int or FrameBound",
+ ));
+ }
+ let n: i64 = match v.extract() {
+ Ok(n) => n,
+ // An int beyond a long is past the thresholds of its side, or else
out of range.
+ Err(_) if start && v.lt(0)? => return
Ok(FrameBound::UnboundedPreceding),
+ Err(_) if !start && v.gt(0)? => return
Ok(FrameBound::UnboundedFollowing),
+ Err(err) => return Err(err),
+ };
Ok(match n {
- UNBOUNDED_PRECEDING => FrameBound::UnboundedPreceding,
- UNBOUNDED_FOLLOWING => FrameBound::UnboundedFollowing,
+ n if start && n <= -i64::MAX => FrameBound::UnboundedPreceding,
+ n if !start && n == i64::MAX => FrameBound::UnboundedFollowing,
0 => FrameBound::CurrentRow,
- n if n < 0 => FrameBound::Preceding(-n),
+ n if n < 0 =>
FrameBound::Preceding(n.checked_neg().unwrap_or(i64::MAX)),
n => FrameBound::Following(n),
})
}
+/// The bounds of a row frame, as PySpark's, whose plan raises
`VALUE_NOT_BETWEEN` for a
+/// boundary of a value that is not an int.
+fn row_frame_bounds(
+ py: Python<'_>,
+ start: &Bound<'_, PyAny>,
+ end: &Bound<'_, PyAny>,
+) -> PyResult<(FrameBound, FrameBound)> {
+ let bounds = (to_frame_bound(start, true)?, to_frame_bound(end, false)?);
+ for (name, bound) in [("start", &bounds.0), ("end", &bounds.1)] {
+ let value = match bound {
+ FrameBound::Preceding(n) => n.saturating_neg(),
+ FrameBound::Following(n) => *n,
+ _ => continue,
+ };
+ if i32::try_from(value).is_err() {
+ return Err(crate::column::raise_pyspark(
+ py,
+ "PySparkValueError",
+ "VALUE_NOT_BETWEEN",
+ &[
+ ("arg_name", name),
+ ("min", &i32::MIN.to_string()),
+ ("max", &i32::MAX.to_string()),
+ ],
+ ));
+ }
+ }
+ Ok(bounds)
+}
+
#[pymethods]
impl PyFrameBound {
fn __repr__(&self) -> String {
@@ -110,13 +189,12 @@ impl PyWindowSpec {
/// Define a ROWS frame between start and end bounds (ints or FrameBounds).
fn rowsBetween(
&self,
+ py: Python<'_>,
start: &Bound<'_, PyAny>,
end: &Bound<'_, PyAny>,
) -> PyResult<PyWindowSpec> {
- let new_spec = self
- .spec
- .clone()
- .rows_between(to_frame_bound(start)?, to_frame_bound(end)?);
+ let (start, end) = row_frame_bounds(py, start, end)?;
+ let new_spec = self.spec.clone().rows_between(start, end);
Ok(PyWindowSpec::new(new_spec))
}
@@ -129,7 +207,7 @@ impl PyWindowSpec {
let new_spec = self
.spec
.clone()
- .range_between(to_frame_bound(start)?, to_frame_bound(end)?);
+ .range_between(to_frame_bound(start, true)?, to_frame_bound(end,
false)?);
Ok(PyWindowSpec::new(new_spec))
}
@@ -181,15 +259,22 @@ impl PyWindow {
/// Create a WindowSpec with a ROWS frame between the given bounds.
#[staticmethod]
- fn rowsBetween(start: &Bound<'_, PyAny>, end: &Bound<'_, PyAny>) ->
PyResult<PyWindowSpec> {
- let spec = WindowSpec::new().rows_between(to_frame_bound(start)?,
to_frame_bound(end)?);
- Ok(PyWindowSpec::new(spec))
+ fn rowsBetween(
+ py: Python<'_>,
+ start: &Bound<'_, PyAny>,
+ end: &Bound<'_, PyAny>,
+ ) -> PyResult<PyWindowSpec> {
+ let (start, end) = row_frame_bounds(py, start, end)?;
+ Ok(PyWindowSpec::new(
+ WindowSpec::new().rows_between(start, end),
+ ))
}
/// Create a WindowSpec with a RANGE frame between the given bounds.
#[staticmethod]
fn rangeBetween(start: &Bound<'_, PyAny>, end: &Bound<'_, PyAny>) ->
PyResult<PyWindowSpec> {
- let spec = WindowSpec::new().range_between(to_frame_bound(start)?,
to_frame_bound(end)?);
+ let spec = WindowSpec::new()
+ .range_between(to_frame_bound(start, true)?, to_frame_bound(end,
false)?);
Ok(PyWindowSpec::new(spec))
}
diff --git a/crates/spark-connect/src/expression.rs
b/crates/spark-connect/src/expression.rs
index d2774da..4c3b1ba 100644
--- a/crates/spark-connect/src/expression.rs
+++ b/crates/spark-connect/src/expression.rs
@@ -1239,8 +1239,10 @@ impl WindowExpressionWrapper {
if let Some((frame_type, lower, upper)) = &self.frame_spec {
let mut frame = proto::expression::window::WindowFrame::default();
frame.frame_type = *frame_type as i32;
- frame.lower = Some(Box::new(to_proto_frame_boundary(lower)));
- frame.upper = Some(Box::new(to_proto_frame_boundary(upper)));
+ // A frame type of 1 is a row frame, and another a range frame.
+ let rows = *frame_type == 1;
+ frame.lower = Some(Box::new(to_proto_frame_boundary(lower, rows)));
+ frame.upper = Some(Box::new(to_proto_frame_boundary(upper, rows)));
window.frame_spec = Some(Box::new(frame));
}
@@ -1249,12 +1251,28 @@ impl WindowExpressionWrapper {
}
}
-/// Convert FrameBoundary to proto FrameBoundary.
+/// Convert FrameBoundary to proto FrameBoundary, as PySpark Connect's
`WindowExpression`
+/// sends the boundary of a value: an integer literal in a row frame, and a
long literal in a
+/// range frame.
fn to_proto_frame_boundary(
boundary: &FrameBoundary,
+ rows: bool,
) -> proto::expression::window::window_frame::FrameBoundary {
- let mut proto_boundary =
proto::expression::window::window_frame::FrameBoundary::default();
- proto_boundary.boundary = match boundary {
+ let offset = |value: i64| {
+ // A ROWS-frame offset must be an IntegerType literal (Spark rejects
bigint with
+ // SPECIFIED_WINDOW_FRAME_UNACCEPTED_TYPE); one out of its range is
sent as a long,
+ // for the server to reject, rather than truncated.
+ let literal = match i32::try_from(value) {
+ Ok(value) if rows => LiteralExpression::int(value),
+ _ => LiteralExpression::long(value),
+ };
+ Some(
+
proto::expression::window::window_frame::frame_boundary::Boundary::Value(Box::new(
+ Expression::literal(literal).to_proto(),
+ )),
+ )
+ };
+ let boundary = match boundary {
FrameBoundary::UnboundedPreceding => {
Some(proto::expression::window::window_frame::frame_boundary::Boundary::Unbounded(true))
}
@@ -1264,20 +1282,11 @@ fn to_proto_frame_boundary(
FrameBoundary::CurrentRow => Some(
proto::expression::window::window_frame::frame_boundary::Boundary::CurrentRow(true),
),
- // A ROWS-frame offset must be an IntegerType literal (Spark rejects
bigint with
- // SPECIFIED_WINDOW_FRAME_UNACCEPTED_TYPE), and preceding is a
negative offset.
- FrameBoundary::Preceding(n) => Some(
-
proto::expression::window::window_frame::frame_boundary::Boundary::Value(Box::new(
- Expression::literal(LiteralExpression::int(-(*n as
i32))).to_proto(),
- )),
- ),
- FrameBoundary::Following(n) => Some(
-
proto::expression::window::window_frame::frame_boundary::Boundary::Value(Box::new(
- Expression::literal(LiteralExpression::int(*n as
i32)).to_proto(),
- )),
- ),
+ // A preceding boundary is a negative offset.
+ FrameBoundary::Preceding(n) => offset(n.saturating_neg()),
+ FrameBoundary::Following(n) => offset(*n),
};
- proto_boundary
+ proto::expression::window::window_frame::FrameBoundary { boundary }
}
impl SortOrder {
diff --git a/crates/spark-connect/src/window.rs
b/crates/spark-connect/src/window.rs
index 9672068..901a9e4 100644
--- a/crates/spark-connect/src/window.rs
+++ b/crates/spark-connect/src/window.rs
@@ -243,6 +243,52 @@ mod tests {
);
}
+ #[test]
+ fn frame_boundaries_are_sent_as_pyspark_sends_them() {
+ use spark_connect_proto::expression::literal::LiteralType;
+ use
spark_connect_proto::expression::window::window_frame::frame_boundary::Boundary;
+ use spark_connect_proto::expression::ExprType;
+
+ let literals = |spec: WindowSpec| {
+ let column =
crate::functions::sum(crate::column::col("id")).over(spec);
+ let Some(ExprType::Window(window)) =
column.expression().to_proto().expr_type else {
+ panic!("expected a window expression");
+ };
+ let frame = window.frame_spec.expect("a frame");
+ [frame.lower.unwrap(), frame.upper.unwrap()].map(|bound| match
bound.boundary {
+ Some(Boundary::Value(value)) => match value.expr_type {
+ Some(ExprType::Literal(literal)) => literal.literal_type,
+ other => panic!("expected a literal, got {other:?}"),
+ },
+ other => panic!("expected a value, got {other:?}"),
+ })
+ };
+ // As PySpark's: the long literals of a range frame, of values out of
the range of an
+ // int too, not truncated.
+ assert_eq!(
+ literals(
+ WindowSpec::new()
+ .range_between(FrameBound::Preceding(1 << 40),
FrameBound::Following(1))
+ ),
+ [
+ Some(LiteralType::Long(-(1 << 40))),
+ Some(LiteralType::Long(1))
+ ]
+ );
+ // The int literals of a row frame, and a long for a value out of the
range of an int,
+ // for the server to reject.
+ assert_eq!(
+ literals(
+ WindowSpec::new()
+ .rows_between(FrameBound::Preceding(1),
FrameBound::Following(1 << 40))
+ ),
+ [
+ Some(LiteralType::Integer(-1)),
+ Some(LiteralType::Long(1 << 40))
+ ]
+ );
+ }
+
#[test]
fn frame_bound_constants() {
assert_eq!(
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 2b27cee..6261cd2 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -1451,6 +1451,28 @@ def
test_functions_send_their_optional_arguments_as_pyspark(session):
assert nulls.groupBy("g").agg(F.first("v", ignorenulls=True)).first()[1]
== 5
+def test_window_frames_return_the_rows_of_pyspark(session):
+ # As PySpark's: a range frame of a boundary out of the range of an int,
which was
+ # truncated, and the frames of the bounds of -sys.maxsize and sys.maxsize,
unbounded.
+ import sys
+
+ from pyspark.errors import PySparkValueError
+ from pyspark.sql import Window
+
+ df = session.range(4)
+ o = Window.orderBy("id")
+
+ def sums(window):
+ return sorted(r.s for r in
df.select(F.sum("id").over(window).alias("s")).collect())
+
+ assert sums(o.rangeBetween(-(1 << 40), 0)) == [0, 1, 3, 6]
+ assert sums(o.rangeBetween(0, 1 << 40)) == [3, 5, 6, 6]
+ assert sums(o.rowsBetween(-sys.maxsize, sys.maxsize)) == [6, 6, 6, 6]
+ assert sums(o.rangeBetween(-sys.maxsize, sys.maxsize)) == [6, 6, 6, 6]
+ with pytest.raises(PySparkValueError, match="VALUE_NOT_BETWEEN"):
+ sums(o.rowsBetween(-(1 << 31) - 1, 0))
+
+
def test_maps_keep_their_entries_in_order_with_keys_of_their_type(session):
row = session.range(1).selectExpr(
"map(2, 'b', 1, 'a') AS ints",
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 8db56e2..a24f545 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -7305,6 +7305,45 @@ def test_window_and_lambda_reprs_as_pyspark():
repr(F.transform("a", lambda x: x + 1)))
+def test_window_frames_and_columns_are_checked_as_pyspark():
+ # As PySpark's: a start of -sys.maxsize or less is unbounded preceding and
an end of
+ # sys.maxsize or more unbounded following; a row frame raises
VALUE_NOT_BETWEEN for a
+ # boundary out of the range of an int, as a start of
Window.unboundedFollowing; and the
+ # columns are Columns or column names, else NOT_EXPECTED_TYPE of `col`.
+ import sys
+
+ from pyspark.errors import PySparkTypeError, PySparkValueError
+ from pyspark.sql import Window
+
+ o = Window.orderBy("id")
+ assert repr(o.rowsBetween(-sys.maxsize, sys.maxsize)) == (
+ "WindowSpec(OrderBy(id ASC NULLS FIRST), "
+ "WindowFrame(ROW_FRAME, -9223372036854775808, 9223372036854775807))")
+ assert repr(Window.rangeBetween(-2 ** 70, 2 ** 70)) == (
+ "WindowSpec(WindowFrame(RANGE_FRAME, -9223372036854775808,
9223372036854775807))")
+ assert repr(o.rangeBetween(-(1 << 40), 0)) == (
+ "WindowSpec(OrderBy(id ASC NULLS FIRST), WindowFrame(RANGE_FRAME,
-1099511627776, 0))")
+ for start, end, name in ((-(1 << 31) - 1, 0, "start"), (0, 1 << 31, "end"),
+ (Window.unboundedFollowing, 0, "start"),
+ (0, Window.unboundedPreceding, "end")):
+ with pytest.raises(PySparkValueError) as error:
+ o.rowsBetween(start, end)
+ assert error.value.getCondition() == "VALUE_NOT_BETWEEN"
+ assert error.value.getMessageParameters() == {
+ "arg_name": name, "min": "-2147483648", "max": "2147483647"}
+ with pytest.raises(AssertionError):
+ o.rowsBetween(1.5, 2)
+ for call, arg_type in ((lambda: Window.partitionBy(1), "int"),
+ (lambda: Window.partitionBy(("id",)), "tuple"),
+ (lambda: Window.orderBy(1), "int"),
+ (lambda: o.orderBy(1.5), "float")):
+ with pytest.raises(PySparkTypeError) as error:
+ call()
+ assert error.value.getMessageParameters() == {
+ "expected_type": "Column or str", "arg_name": "col", "arg_type":
arg_type}
+ assert repr(Window.partitionBy(["a", "b"])) == "WindowSpec(PartitionBy(a,
b))"
+
+
def test_expressions_carry_their_origins_as_pyspark():
# As PySpark's _with_origin: col() and the Column methods give the
expressions that they
# build the origin of their name and their call site in user code, as the
python_origin of
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]