[
https://issues.apache.org/jira/browse/SPARK-61044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61044:
-----------------------------------
Labels: pull-request-available (was: )
> Send the frame boundaries of rowsBetween and rangeBetween, and check the
> columns of Window, in the Python client as PySpark does
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61044
> URL: https://issues.apache.org/jira/browse/SPARK-61044
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{WindowSpec.rangeBetween(start, end)}} sends the
> boundaries of its frame as long literals, so that
> {{Window.orderBy("id").rangeBetween(-(1 << 40), 0)}} is the frame of every
> preceding value. {{rowsBetween}} sends them as integer literals, and raises
> {{VALUE_NOT_BETWEEN}} for one outside of the range of an int. Both take a
> {{start}} of {{-sys.maxsize}} or less as {{Window.unboundedPreceding}}, and
> an {{end}} of {{sys.maxsize}} or more as {{Window.unboundedFollowing}}, and a
> {{start}} of {{Window.unboundedFollowing}}, or an {{end}} of
> {{Window.unboundedPreceding}}, is a value, not unbounded.
> {{Window.partitionBy}} and {{orderBy}} raise {{NOT_EXPECTED_TYPE}} of the
> argument {{col}} for a column that is neither a Column nor a column name.
> The Rust-backed Python client sends the boundaries as integer literals of
> their values truncated to an int: {{rangeBetween(-(1 << 40), 0)}} is the
> frame of the current row only, which gives wrong results. It raises a
> TypeError for {{rowsBetween(-sys.maxsize, sys.maxsize)}} and for a boundary
> outside of the range of an int, takes {{Window.unboundedFollowing}} as a
> start and {{Window.unboundedPreceding}} as an end as unbounded, and names the
> argument {{cols}}. Send the frames, and check the columns, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
