HyukjinKwon opened a new pull request, #245: URL: https://github.com/apache/spark-connect-rust/pull/245
### What changes were proposed in this pull request? [SPARK-61046](https://issues.apache.org/jira/browse/SPARK-61046): `GroupedData.pivot` now pivots only a `groupBy`, and checks its arguments, as PySpark Connect does. - The pivot of a `rollup`, a `cube` or `groupingSets` raises `UNSUPPORTED_OPERATION` of "PIVOT after ROLLUP", "PIVOT after CUBE" or "PIVOT after GROUPING_SETS". The pivot of a pivot raises it of "Repeated PIVOT operation". As in PySpark, the grouping is checked before the arguments. - It then raises `NOT_EXPECTED_TYPE`: - of `pivot_col`, for a pivot column that is not a str - of `values`, for values that are not a list, such as a tuple - of `value`, for a value that is not a bool, a float, an int or a str The core `GroupedData::pivot` now returns a `Result`, with the error of the new `GroupedData::check_pivot`. ### Why are the changes needed? The client pivoted any grouping as a `groupBy`. So `df.rollup("k").pivot("s").sum("v")` returned the rows of `df.groupBy("k").pivot("s").sum("v")`, without the rows of the rollup, and a second pivot replaced the first. These wrong results are the main fix. It also raised `TypeError`s for the arguments, and took a tuple of values. ### Does this PR introduce _any_ user-facing change? Yes. `pivot` raises the errors of PySpark 4.3.0, rather than returning the rows of another grouping. ### How was this patch tested? - A new live test checks the `UNSUPPORTED_OPERATION` of the pivots of a rollup, a cube, grouping sets and a pivot, the rows of a pivot of a `groupBy`, and the `NOT_EXPECTED_TYPE` of a tuple of values. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks these errors and their parameters, and that the grouping is checked first. - A new lib test checks the errors of the core `pivot`. - On a local server, 70 cases of `pivot` give the rows and errors of official PySpark 4.3.0 for `groupBy`, `rollup`, `cube` and `groupingSets`. - The lib tests of both crates (575 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1811 passed, 8 skipped), and the live drop-in and Arrow suites (110 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
