HyukjinKwon opened a new pull request, #239: URL: https://github.com/apache/spark-connect-rust/pull/239
### What changes were proposed in this pull request? [SPARK-61037](https://issues.apache.org/jira/browse/SPARK-61037): `groupingSets` now groups by its grouping columns `*cols`, as PySpark Connect does. `df.groupingSets(groupingSets, *cols)` groups by the grouping sets with the grouping columns `cols`, which are the columns of the result besides the aggregations. A grouping column that a set does not hold is null in the rows of that set, and without `cols` the result has only the aggregations. As PySpark's `_to_cols` of ordinals: - a set is a sequence of Columns, column names and 1-based ordinals, or one of them alone - `cols` are such columns themselves, not a list of them - anything else raises `NOT_EXPECTED_TYPE` The core takes the grouping columns in the new `DataFrame::grouping_sets_with_columns`. ### Why are the changes needed? The client ignored `cols` and grouped by the union of the columns of the sets. `df.groupingSets([["a"]], "a", "b").agg(sum("c"))` had the columns `a` and the sum, not `a`, `b` and the sum. `df.groupingSets([["a"], ["b"]]).agg(sum("c"))` had `a`, `b` and the sum, not the sum only. ### Does this PR introduce _any_ user-facing change? Yes. `groupingSets` returns the columns and rows of PySpark 4.3.0. ### How was this patch tested? - A new live test checks the columns and rows of `groupingSets` with and without grouping columns, and of ordinals. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks the `Aggregate` that `groupingSets` sends to a mock Connect server, and its errors. - On a local server, 14 cases of `groupingSets` behave as in official PySpark 4.3.0. - The lib tests of both crates (571 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1811 passed, 8 skipped), and the live drop-in and Arrow suites (109 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
