andygrove opened a new issue, #339: URL: https://github.com/apache/datafusion-spark/issues/339
### Is your feature request related to a problem or challenge? `SparkCollectList` and `SparkCollectSet` (`crates/spark/src/function/aggregate/collect.rs`) implement only `accumulator()`, so grouped aggregation falls back to `GroupsAccumulatorAdapter`: one boxed `Accumulator` per group. At high grouping cardinality this dominates. On 2M rows grouped by a string key with 200k distinct values, `collect_list` ran at 0.49x of Spark's JVM implementation, while `count(*)` with the same grouping ran at 2.88x. Comet works around this with its own `GroupsAccumulator`s (apache/datafusion-comet#5803), which were 85–99% faster than the adapter at 16k groups. Moved from apache/datafusion#25118, which keeps the `array_agg` accumulator work. ### Describe the solution you'd like - `SparkCollectList`: implement `create_groups_accumulator` by reusing `ArrayAggGroupsAccumulator` with `ignore_nulls = true`, turning the lists it leaves null into empty lists (Spark returns `[]`, not `NULL`, for a group whose inputs were all NULL). - `SparkCollectSet`: implement a grouped accumulator. apache/datafusion#26357 added a `GroupsAccumulator` for `array_agg(DISTINCT)` that may be reusable here. Comet's implementations in `native/spark-expr/src/agg_funcs/collect.rs` (apache/datafusion-comet#5803) could be upstreamed more or less as they are. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
