adiranjan25 opened a new pull request, #59981: URL: https://github.com/apache/spark/pull/59981
### What changes were proposed in this pull request? This PR fixes named `size` aggregation in the pandas API on Spark. Previously, `groupby().agg()` with a named `size` aggregation generated the Spark SQL expression `size(column)`. Since Spark SQL's `size()` function expects an array or map, this caused a datatype mismatch for ordinary numeric columns. The fix translates named `size` aggregation to `count(1)`, preserving pandas semantics by counting all rows, including rows with null values. ### Why are the changes needed? In pandas, `GroupBy.size()` counts all rows in each group, whereas `GroupBy.count()` excludes null values. Named aggregations should preserve this distinction. ### Does this PR introduce any user-facing change? Yes. Named `size` aggregation now works correctly for ordinary columns in the pandas API on Spark. ### How was this patch tested? Added `test_aggregate_relabel_size`, comparing pandas and pandas-on-Spark results for named `size` and `count` aggregations on data containing null values. The regression test failed before the fix and passed afterward. All three targeted aggregation tests passed locally. Ruff formatting and Git whitespace checks also passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
