HyukjinKwon opened a new pull request, #228: URL: https://github.com/apache/spark-connect-rust/pull/228
### What changes were proposed in this pull request? [SPARK-61028](https://issues.apache.org/jira/browse/SPARK-61028): the aggregations of `GroupedData` and `DataFrame.agg` now name and take their columns as PySpark's do. - `df.groupBy(...).count()` names its column `count`, as it aggregates `count(lit(1)).alias("count")`. - `sum`, `avg`, `mean`, `max` and `min` of a GroupedData aggregate every numeric column of the DataFrame when no column is given. For a column that is not a numeric one of the schema, they raise `NOT_NUMERIC_COLUMNS`, as PySpark's `verify_numeric_col_name` finds it from the parts of the name. They assert that the columns are strings. - `GroupedData.agg` asserts that it is given expressions, which are Columns unless they are one dict. - `df.agg` raises `CANNOT_BE_EMPTY` without expressions, and aggregates a dict of column names and functions. It asserts that other expressions are Columns. In the Rust API, `GroupedData::numeric_agg` aggregates numeric columns as PySpark's `_numeric_agg`, and `GroupedData::count` names its column `count`. ### Why are the changes needed? - The column of `count()` was named `count(1)`, so `df.groupBy("k").count().orderBy("count")` failed. - `sum()` and the others aggregated no column when none was given. - `df.groupBy("k").sum("k")` failed on the server. - `df.agg({"x": "max"})` raised a `TypeError`. ### Does this PR introduce _any_ user-facing change? Yes. These aggregations name their columns, and take their arguments, as PySpark 4.3.0 does. ### How was this patch tested? - A new live test checks the names and the results of these aggregations, and `NOT_NUMERIC_COLUMNS`. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the aggregations sent to a mock Connect server and the errors. It fails without this change. - A new lib test checks the parsing of attribute names and the numeric check of nested fields, against PySpark's `parse_attr_name`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
