naman-modi opened a new pull request, #23838: URL: https://github.com/apache/datafusion/pull/23838
## Which issue does this PR close? - Part of #22710. ## Rationale for this change There's almost no slt coverage for grouped aggregation under a memory limit, where the aggregate spills to disk and re-groups the spilled state. #23657 covers the ordered path & this covers the unordered/hash path. ## What changes are included in this PR? - Adds `aggregate_memory_limit.slt` in which the high-cardinality "GROUP BY" is run under a 1M limit, so the hash aggregate spills. The group key is scrambled with `(v * 7) % 100000` (a bijection, so still 100000 groups) to keep the input unsorted; otherwise it takes the streaming path and never spills. - Scoped to a single partition (`target_partitions = 1`), so the spill happens in one aggregate operator with no repartition. - Cases cover different accumulator states: single-column, multi-column, count(DISTINCT), sum/min/max, avg (widening), array_agg (growable). ## Are these changes tested? This PR is tests. All pass locally. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
