Hyukjin Kwon created SPARK-61044:
------------------------------------
Summary: Send the frame boundaries of rowsBetween and
rangeBetween, and check the columns of Window, in the Python client as PySpark
does
Key: SPARK-61044
URL: https://issues.apache.org/jira/browse/SPARK-61044
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{WindowSpec.rangeBetween(start, end)}} sends the
boundaries of its frame as long literals, so that
{{Window.orderBy("id").rangeBetween(-(1 << 40), 0)}} is the frame of every
preceding value. {{rowsBetween}} sends them as integer literals, and raises
{{VALUE_NOT_BETWEEN}} for one outside of the range of an int. Both take a
{{start}} of {{-sys.maxsize}} or less as {{Window.unboundedPreceding}}, and an
{{end}} of {{sys.maxsize}} or more as {{Window.unboundedFollowing}}, and a
{{start}} of {{Window.unboundedFollowing}}, or an {{end}} of
{{Window.unboundedPreceding}}, is a value, not unbounded.
{{Window.partitionBy}} and {{orderBy}} raise {{NOT_EXPECTED_TYPE}} of the
argument {{col}} for a column that is neither a Column nor a column name.
The Rust-backed Python client sends the boundaries as integer literals of their
values truncated to an int: {{rangeBetween(-(1 << 40), 0)}} is the frame of the
current row only, which gives wrong results. It raises a TypeError for
{{rowsBetween(-sys.maxsize, sys.maxsize)}} and for a boundary outside of the
range of an int, takes {{Window.unboundedFollowing}} as a start and
{{Window.unboundedPreceding}} as an end as unbounded, and names the argument
{{cols}}. Send the frames, and check the columns, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
