QinXi-ai opened a new pull request, #27299: URL: https://github.com/apache/datafusion/pull/27299
## Which issue does this PR close? Part of #25936. This implements the accumulator step in the follow-up plan. ## Rationale for this change The native grouped integer `COUNT(DISTINCT)` accumulator currently probes the shared distinct hash set for every non-null input row. For Q10, which has a million groups with four copies of one value per group, keeping that value inline avoids those probes. I built both versions from `c8265b68` on a shared Linux host, using a Xeon Gold 6238T, Rust 1.99.1 and `release-nonlto`. The suite used four million rows, one target partition, greedy / 1 GiB, with `single_distinct_prefer_native` enabled. These are medians from three alternating rounds, six iterations per round with the first discarded, giving 15 retained samples per version. CPU affinity was 0-7, with builds stopped during timing. | Case | Main | This PR | Elapsed change | Peak pool memory, main / PR | | --- | ---: | ---: | ---: | ---: | | Q01, 2,000 groups, high NDV | 318.93 ms | 303.87 ms | -4.7% | 168.17 / 168.17 MiB | | Q02, 2,000 groups, eight distinct values each | 79.11 ms | 80.65 ms | +1.9% | 0.824 / 0.511 MiB | | Q07, skewed integer groups | 208.60 ms | 203.97 ms | -2.2% | 42.07 / 42.07 MiB | | Q09, skewed composite groups | 271.62 ms | 269.60 ms | -0.7% | 42.25 / 42.25 MiB | | Q10, a million groups, one distinct value each | 382.14 ms | 218.90 ms | -42.7% | 95.88 / 61.88 MiB | | Q11, a million groups, at most two values each | 417.89 ms | 317.86 ms | -23.9% | 95.88 / 82.88 MiB | | Q12, 95% of groups have only NULL arguments | 677.61 ms | 675.68 ms | -0.3% | 55.19 / 61.88 MiB | The inline buffer allocates a typed slot for every group, including groups with only NULL arguments. Q12 exposes that cost, with about 6.7 MiB more peak pool memory. ## What changes are included in this PR? Keep each group's first non-null value in a typed vector and put additional values in the shared hash set. After a batch where the average number of additional distinct values per group exceeds eight, move the inline values into the set and release the vector. Subsequent batches use the hash-only loop. This avoids the per-row cache check for the large sets in Q07 and Q09. CPU-0 repeats measured -2.9% and -3.5% elapsed changes for those two queries. Both modes use the existing List state format. State emission, partial emission, merge and memory accounting handle the inline values and the switch to the hash-only mode. Add Q11 and Q12 to the SQL benchmark suite to cover small distinct sets and NULL-heavy groups. ## What is the testing strategy for this PR? Added unit tests for partial emission, overlapping merges, group-index shifts, promotion, buffer accounting and reuse. SQL tests exercise small batches and overlapping partial states through the normal query path. Validated on Linux with Rust 1.99.1. - All 64 aggregate-common tests passed. - The extended workspace suite passed 13,626 Rust tests, with 8 ignored, and completed all 547 SQL test files. - All 116 CLI tests passed. - Formatting, strict all-targets/all-features Clippy and the full `uv run ./dev/rust_lint.sh` suite passed. - All twelve SQL benchmark queries passed their result assertions. ## Are there any user-facing changes? Native grouped integer `COUNT(DISTINCT)` has the runtime and memory changes shown above. Prepared with Codex assistance. Opening as a draft for author review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
