Rachelint opened a new pull request, #26109: URL: https://github.com/apache/datafusion/pull/26109
## Which issue does this PR close? None. Experimental follow-up to #26100; not intended for merge as-is. ## Rationale for this change Measure the incremental benefit of restoring the StringView allocation optimizations from #25959 on top of the isolated Partial flush implementation in #26100. The direct baseline is `634a59fff53e19c630657d728c4db3a57204727e`, with `partial_aggregation_flush_bytes=2097152` and `partial_aggregation_flush_rows=0`. Main is pinned to `7b1e55d4c2291acf6b9e7eb20dcdc62f25a8fe98` as a third comparison. The incremental implementation is commit `522c7492f`; earlier commits in this PR are inherited from #26100. ## What changes are included in this PR? - Reuse the allocated hash table for single StringView group keys after a threshold-triggered Partial flush. - Use one payload buffer after that flush, up to the u32 view-offset limit, reserving the previous flush's payload length. - Transfer emitted views and payload buffers to the outgoing array so subsequent batches cannot mutate them. - Keep retained allocations in the existing memory accounting. Memory-pressure emission continues to release allocations. - Document the allocation behavior of the experimental flush setting. The flush thresholds and their defaults are unchanged from #26100. ## What is the testing strategy for this PR? Passed: - `cargo fmt --all` and format check; `git diff --check`. - `cargo check -p datafusion-physical-plan`. - `cargo clippy --all-targets --all-features -- -D warnings`. - All 14 `binary_view_map` tests, including repeated flushes with nulls, inline strings, and long strings crossing the former payload-block boundary; the test verifies retained table capacity, contiguous subsequent output, output ownership, and explicit allocation release. - All 11 `aggregates::hash_stream::tests`, covering memory-pressure output, skip-partial accounting, and spill paths. - `aggregate_partial_flush.slt`, comparing result correctness with disabled, byte-only, and row-only flush settings across string, integer, mixed, nullable, and nested-state aggregates. - Configuration docs regenerated and `./ci/scripts/doc_prettier_check.sh --write --allow-dirty` passed. ECS performance comparison is running: all q0-q42 in `clickbench_partitioned`, 99,997,497 rows, 12 partitions pinned to CPUs 0-11, batch 8192, Rust 1.99.0, `release-nonlto`. Two rounds reverse query/variant order, four iterations per cell with the first discarded, yielding six warm samples per query/version. The three versions are main, #26100 with 2 MiB flushing, and this PR with the same thresholds. Source and binary hashes are recorded. Results will be added when both rounds finish. ## Are there any user-facing changes? When proactive Partial flush is enabled, single StringView group keys retain hash-table capacity and reserve payload storage across threshold flushes. Default configurations keep flushing disabled. This PR includes the two experimental configuration fields introduced in #26100. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
