[ 
https://issues.apache.org/jira/browse/SPARK-61887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon resolved SPARK-61887.
----------------------------------
    Resolution: Fixed

Issue resolved by pull request 794
[https://github.com/apache/spark-connect-rust/pull/794]

> Replace the partitioning and the ordering of a WindowSpec in partitionBy and 
> orderBy as PySpark does, rather than append to them
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61887
>                 URL: https://issues.apache.org/jira/browse/SPARK-61887
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.4.0
>            Reporter: Hyukjin Kwon
>            Assignee: Hyukjin Kwon
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: connect-rust-4.4.0
>
>
> WindowSpec.partitionBy and WindowSpec.orderBy of the drop-in and of the Rust 
> API's WindowSpec::partition_by and order_by append their columns to those of 
> the spec, where PySpark's (connect/window.py:85-96), as Spark's WindowSpec, 
> return a spec whose partitioning or ordering is the new columns only. So a 
> chained call gives other results: of rows (1,10), (2,10), (3,20) of columns a 
> and b, count("*").over(Window.partitionBy("a").partitionBy("b")) gives 2, 2, 
> 1 on PySpark and 1, 1, 1 on the drop-in, and row_number() over 
> Window.orderBy("a").orderBy(col("b").desc(), "a") gives 2, 3, 1 on PySpark 
> and 1, 2, 3 on the drop-in. The repr of the spec differs too (PartitionBy(b) 
> against PartitionBy(a, b)).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to