Rachelint opened a new pull request, #26097: URL: https://github.com/apache/datafusion/pull/26097
## Which issue does this PR close? None. This is a benchmark-only experiment based on #25724 at `88d36d4d285ccdf7da30a852725a75a8700133a8`. ## Rationale for this change Measure whether flushing Partial aggregation with a smaller allocated working set improves end-to-end ClickBench performance. Compare directly with the pinned head of #25724 to isolate the threshold change. ## What changes are included in this PR? The only runtime change relative to the pinned baseline is an additional Partial flush condition: `AggregateHashTable::memory_size() >= 2 * 1024 * 1024`, checked after each input batch. The existing group-count threshold remains an alternative trigger. Reuse the memory size already computed for the reservation. The baseline's recurrence detection and disabling policy, fixed-width eligibility, and Final aggregation behavior are unchanged. In particular, string-key Partial aggregation (including Q33/Q34) remains ineligible, as in the pinned baseline. This PR includes the commits from #25724 because the experiment needs its aggregation implementation. Review the experiment's incremental diff against the pinned SHA. ## What is the testing strategy for this PR? Added a regression test comparing aggregate counts with flushing disabled and with the byte threshold enabled, using fewer groups than the group-count threshold. - `cargo fmt --all` and `cargo fmt --all --check`: passed. - `cargo clippy --all-targets --all-features -- -D warnings`: passed. - Aggregation unit tests from the CI-profile test binary: 300 passed, including the new byte-threshold test. - Required workspace extended test command with `--profile ci` and `avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption`: passed (12,383 Rust tests and 527 SQLLogicTest files). Benchmark: `clickbench_partitioned`, baseline `88d36d4d285ccdf7da30a852725a75a8700133a8`, with `DATAFUSION_EXECUTION_HASH_AGGREGATE_BUCKET_THRESHOLD=262144` on both sides. ## Are there any user-facing changes? Experimental earlier Partial flushing for eligible fixed-width aggregation when bucket aggregation is enabled. No new public configuration or API. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
