xiangfu0 opened a new pull request, #19825: URL: https://github.com/apache/pinot/pull/19825
Unordered single-COUNT group-by combines currently allocate a row array and `Record` for every partial group, even when the group already exists. This change extracts the count before merging and creates the row only for a newly admitted group, preserving atomic updates, group limits and table fallbacks. ### Benchmarks Local synthetic JMH allocation measurements against `f818bae6d75a91228eb3aac95c921bca4aeb2128`, with four merge workers; MB are decimal and use `gc.alloc.rate.norm`. | Workload | Baseline → candidate allocation | Reduction | |---|---:|---:| | 100k inputs / 20k groups (80% duplicates) | 21.016 → 17.174 MB/batch | 18.17–18.28% | | 100k inputs / 1k groups (99% duplicates) | 20.061 → 15.310 MB/batch | 23.68–23.69% | | 100k inputs / 20k groups, 16-key collision families | 26.859 → 23.015 MB/batch | 14.31% | | 1M inputs / 487,467 groups | 228.130 → 203.495 MB/batch | 10.80% | | Full instance query, four copies, StarTree | 76.545–76.632 → 69.107–69.989 MB/query | 8.67–9.72% | | Full instance query, four copies, raw scan | 34.986–35.387 → 27.941–28.755 MB/query | 17.81–21.04% | The first two rows show median allocations and paired reduction ranges across three AB/BA/AB pairs; collision/large-map cases have one AB pair, and full-query modes have two AB/BA pairs. Unique-input combine and one-copy `SimpleIndexedTable` query controls show approximately unchanged allocation. Apple M2 Max, Temurin 25+36, JMH 1.37, one fork/thread, 1 GB G1 heap, `ActiveProcessorCount=4`, two parallel/one concurrent GC workers; combine warmup/measurement: 4×1s/6×1s, full query: 3×1s/5×1s. Combine batches include fresh key/row materialization, merge, finish and traversal; full queries include planning, segment execution, merge and result materialization. CPU and latency results remain inconclusive; allocation volume does not establish reduced retained heap or peak memory. These are synthetic fixtures, not customer capacity results. ### Reproduce the allocation pattern Run COUNT grouped on two columns across segments with overlapping keys, without ORDER BY or grouping sets: ```sql SET numGroupsLimit=2000000; SET maxExecutionThreads=4; SELECT k1, k2, COUNT(*) FROM CountBenchmark GROUP BY k1, k2 LIMIT 2000000; ``` The full-query fixture uses four independently mmap-loaded copies of a 352,832-row INT/INT segment: 192,800 merge inputs, 48,200 groups and 75% duplicates, with null handling off. Complete group counts, result ownership and DataTable wire round trips match in raw and StarTree modes. ### Validation 63 focused test executions passed on the rebased branch without dependency workarounds; Spotless, license formatting, Checkstyle and license validation passed; independent correctness/concurrency review found no issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
