Dandandan opened a new pull request, #24815: URL: https://github.com/apache/datafusion/pull/24815
## Which issue does this PR close? - None; noticed while profiling ClickBench aggregation. ## Rationale for this change Growing one allocation copies everything in it, so high-cardinality groupings spend a large share of their time reallocating per-group state. ## What changes are included in this PR? - `BlockedVec`: per-group state in one `Vec` up to `THRESHOLD_LEN` groups, fixed-size blocks above it, so growth appends a block instead of copying. - Accumulators take `storage_mut()` once per batch instead of dispatching per group, so the flat path stays identical to a plain `Vec`. - Wired into `count`, `sum` (`PrimitiveGroupsAccumulator`) and `avg`. ## Are these changes tested? - Yes: unit tests for growth across the switch, `take_first` shifting, and capacity accounting; existing aggregate tests and `sqllogictest` aggregate files pass. ## Are there any user-facing changes? - No; `GroupsAccumulator::size` now reports block capacity, which is the same quantity measured differently. ## Benchmarks - ClickBench is currently neutral (q32 1.04, q18 0.98, q16 1.02, q33 1.02): blocks remove the growth copies but `emit` still concatenates them. - The win needs per-block emit so a block goes to Arrow as its own buffer; that is the follow-up this PR is the foundation for. - Measured cost of the extra load, with the check hoisted out of the loop: 1.02x below the threshold, 1.49x above it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
