Hyukjin Kwon created SPARK-61037:
------------------------------------
Summary: Group by the grouping columns of groupingSets in the
Python client as PySpark does
Key: SPARK-61037
URL: https://issues.apache.org/jira/browse/SPARK-61037
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{df.groupingSets(groupingSets, *cols)}} groups by the
grouping sets {{groupingSets}} with the grouping columns {{cols}}, which are
the columns of the output besides the aggregations. {{df.groupingSets([["a"]],
"a", "b").agg(sum("c"))}} has the columns {{a}}, {{b}} and the sum, {{b}} null,
and {{df.groupingSets([["a"], ["b"]]).agg(sum("c"))}} only the sum.
The Rust-backed Python client ignores {{cols}} and groups by the columns of the
sets: the columns of the first are {{a}} and the sum, and those of the second
{{a}}, {{b}} and the sum. Group as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
