Hyukjin Kwon created SPARK-61887:
------------------------------------
Summary: Replace the partitioning and the ordering of a WindowSpec
in partitionBy and orderBy as PySpark does, rather than append to them
Key: SPARK-61887
URL: https://issues.apache.org/jira/browse/SPARK-61887
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
WindowSpec.partitionBy and WindowSpec.orderBy of the drop-in and of the Rust
API's WindowSpec::partition_by and order_by append their columns to those of
the spec, where PySpark's (connect/window.py:85-96), as Spark's WindowSpec,
return a spec whose partitioning or ordering is the new columns only. So a
chained call gives other results: of rows (1,10), (2,10), (3,20) of columns a
and b, count("*").over(Window.partitionBy("a").partitionBy("b")) gives 2, 2, 1
on PySpark and 1, 1, 1 on the drop-in, and row_number() over
Window.orderBy("a").orderBy(col("b").desc(), "a") gives 2, 3, 1 on PySpark and
1, 2, 3 on the drop-in. The repr of the spec differs too (PartitionBy(b)
against PartitionBy(a, b)).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]