Hyukjin Kwon created SPARK-61121:
------------------------------------
Summary: Take a float frame bound past its threshold as unbounded
in rowsBetween and rangeBetween, in the Python client as PySpark does
Key: SPARK-61121
URL: https://issues.apache.org/jira/browse/SPARK-61121
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark's {{WindowSpec.rowsBetween}} and {{rangeBetween}} compare their bounds
with {{Window._PRECEDING_THRESHOLD}} and {{Window._FOLLOWING_THRESHOLD}} before
{{WindowFrame}} asserts an int. A start at or below the first is unbounded
preceding, and an end at or above the second is unbounded following, also for a
float. So {{w.rowsBetween(float("-inf"), float("inf"))}} is a frame of
unbounded bounds, as Apache Spark's {{test_window_functions}} uses it.
The Python client raises {{AssertionError}} for any float bound, also an
infinite one.
The fix is to take a float bound past its threshold as unbounded, as PySpark
does, and to raise {{AssertionError}} for other floats.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
