zhuqi-lucas opened a new issue, #26347: URL: https://github.com/apache/datafusion/issues/26347
### Is your feature request related to a problem or challenge? `array_agg(DISTINCT x)` has no `GroupsAccumulator`. `groups_accumulator_supported` returns false whenever `is_distinct` is set (`datafusion/functions-aggregate/src/array_agg.rs`), so grouped queries fall back to `GroupsAccumulatorAdapter`. That adapter creates one `DistinctArrayAggAccumulator` per group, and each one carries its own `HashSet<ScalarValue>`. With many groups that each hold a small distinct list, the per-group boxes and hash sets dominate both memory and CPU. We hit this with a daily rollup of the shape `GROUP BY date, account, endpoint, status ... array_agg(DISTINCT ticker)`. In a heap profile of that query, `DistinctArrayAggAccumulator` was 2.4 GB of live memory. `SingleDistinctToGroupBy` does not cover it once the query also has a `FILTER` or other aggregates such as `count(*)`. ### Describe the solution you'd like Add a `GroupsAccumulator` for `array_agg(DISTINCT)`, following `PrimitiveDistinctCountGroupsAccumulator` (`count_distinct/groups.rs`). That accumulator keeps one shared `HashSet<(group_idx, value)>` for all groups instead of a set per group. For the list output: * Intern values once, so each distinct value is stored a single time across groups. * Dedup on `(group_idx, value_id)`. * In `evaluate`, build the `ListArray` from the deduplicated pairs grouped by `group_idx`. * Implement `state` / `merge_batch` / `convert_to_state` / `size` so spilling and memory accounting work. Start with primitive and string types. Keep the current path for `ORDER BY` inside the aggregate and for other types. ### Describe alternatives you've considered Rewriting the query as a two-level aggregation works today: first `SELECT DISTINCT keys, x`, then a non-distinct `array_agg(x)` per group, joined back to the other aggregates. It is easy to miss and needs an extra join. ### Additional context A string `count(DISTINCT)` groups accumulator could share the same interning approach. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
