bhonepyisone opened a new pull request, #23180: URL: https://github.com/apache/datafusion/pull/23180
## What Fixes the hash aggregate output performance regression introduced by #23055. ## Problem PR #23055 introduced `EmitTo::First(batch_size)` for incremental hash aggregate output. During the **terminal output phase** (`Outputting` state), no more `intern()` calls happen, but `emit(EmitTo::First(n))` still maintained the hash lookup state on **every output batch**: | Implementation | Wasted work per batch | |---|---| | `GroupValuesColumn` (multi-col) | `map.retain()` with complex index-list manipulation | | `GroupValuesPrimitive` (single-col) | `map.retain()` shifting all remaining indexes | | `GroupValuesRows` (fallback) | Rebuilds rows + `map.retain()` | This caused **O(groups × output_batches)** unnecessary work. ## Performance Impact On TPC-DS SF10 query 23 with 24 target partitions: - **Before #23055**: ~2.43s mean - **After #23055 (regressed)**: ~8.04s mean - **With this fix**: should return to ~2.43s The problematic node spent ~32s in `emitting_time` vs ~3s in actual aggregation time. ## Fix Added `release_interning_state()` to the `GroupValues` trait, called once during the `Building → Outputting` state transition in `start_outputting()`. It clears the hash map and interning buffers, making `retain` a no-op during terminal output. The group values themselves are stored separately (in `values` / `group_values` / column builders) and are **not** cleared, so output data is preserved. ## Files Changed - `group_values/mod.rs` — Added `release_interning_state()` to the `GroupValues` trait - `aggregate_hash_table/common.rs` — Call it in `start_outputting()` - `multi_group_by/mod.rs` — Clear map, hashes, group-index lists (the main hot path) - `row.rs` — Clear map, hashes, row buffers - `single_group_by/primitive.rs` — Clear map - `single_group_by/boolean.rs` — No-op (no hash map) - `single_group_by/bytes.rs` — No-op (map rebuilt on emit) - `single_group_by/bytes_view.rs` — No-op (map rebuilt on emit) Closes #23178 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
