[
https://issues.apache.org/jira/browse/SPARK-61037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61037:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Group by the grouping columns of groupingSets in the Python client as PySpark
> does
> ----------------------------------------------------------------------------------
>
> Key: SPARK-61037
> URL: https://issues.apache.org/jira/browse/SPARK-61037
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{df.groupingSets(groupingSets, *cols)}} groups by the
> grouping sets {{groupingSets}} with the grouping columns {{cols}}, which are
> the columns of the output besides the aggregations.
> {{df.groupingSets([["a"]], "a", "b").agg(sum("c"))}} has the columns {{a}},
> {{b}} and the sum, {{b}} null, and {{df.groupingSets([["a"],
> ["b"]]).agg(sum("c"))}} only the sum.
> The Rust-backed Python client ignores {{cols}} and groups by the columns of
> the sets: the columns of the first are {{a}} and the sum, and those of the
> second {{a}}, {{b}} and the sum. Group as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
