[
https://issues.apache.org/jira/browse/SPARK-61887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61887.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 794
[https://github.com/apache/spark-connect-rust/pull/794]
> Replace the partitioning and the ordering of a WindowSpec in partitionBy and
> orderBy as PySpark does, rather than append to them
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61887
> URL: https://issues.apache.org/jira/browse/SPARK-61887
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> WindowSpec.partitionBy and WindowSpec.orderBy of the drop-in and of the Rust
> API's WindowSpec::partition_by and order_by append their columns to those of
> the spec, where PySpark's (connect/window.py:85-96), as Spark's WindowSpec,
> return a spec whose partitioning or ordering is the new columns only. So a
> chained call gives other results: of rows (1,10), (2,10), (3,20) of columns a
> and b, count("*").over(Window.partitionBy("a").partitionBy("b")) gives 2, 2,
> 1 on PySpark and 1, 1, 1 on the drop-in, and row_number() over
> Window.orderBy("a").orderBy(col("b").desc(), "a") gives 2, 3, 1 on PySpark
> and 1, 2, 3 on the drop-in. The repr of the spec differs too (PartitionBy(b)
> against PartitionBy(a, b)).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]