xiangfu0 opened a new pull request, #19264: URL: https://github.com/apache/pinot/pull/19264
## Summary GROUP BY `GROUPING SETS` / `ROLLUP` / `CUBE` previously expanded every input row into one group per grouping set in each segment, so the per-set fan-out cost scaled with the number of scanned rows (`O(rows * numSets)`). This makes ROLLUP/CUBE several times slower than a plain GROUP BY over the same columns, even though the results are derivable from a single base grouping. This PR aggregates only the **base grouping** (the union of all grouping-set columns) once per segment — reusing the fast plain-GROUP-BY path — then **derives** the individual grouping-set records from those base groups. The per-set fan-out moves from `O(rows)` to `O(base groups)`. ## Approach For each base group, project it into each grouping set: rolled-up (non-participating) columns become `NULL`, the `$groupingId` discriminator is stamped, and the base group's aggregation intermediates are merged into the derived group. This reuses the existing `AggregationFunction#merge` machinery that the combine/reduce phases already rely on, so it is exact for every mergeable aggregation. Because a base group's intermediate flows into every grouping set and `merge` mutates/returns its argument, each base intermediate is **cloned per set** (via the function's serialize/deserialize round-trip; scalar intermediates are immutable and skipped) before it can become a merge target, keeping object-backed accumulators (AVG, DISTINCTCOUNT, percentiles, ...) correct. The base path also applies the same per-set bucketed segment trim as the expansion path. ## Behavior change The base-aggregation path is **enabled by default** via a new `groupingSetsBaseAggregation` query option. Set `groupingSetsBaseAggregation=false` to force the legacy per-row expansion path. Results are identical to the expansion path (verified by equivalence tests). Carve-outs that fall back to the expansion path: - **Multi-value group-by columns** — an MV column fans a row across its values in the base grouping, which would over-count when that column is rolled up. - **Filtered aggregations** — these share a single group-key generator across aggregation groups via a distinct segment path. ## Also included Improvements to the legacy expansion generator (used for the fallback cases and `groupingSetsBaseAggregation=false`): - Resolves dictionary-encoded columns via native dict-ids instead of re-hashing raw values. - Packs composite keys into a primitive `long` when they fit, using a `Long2IntOpenHashMap` instead of `Object2IntOpenHashMap<FixedIntArray>` (avoids per-group object allocation and array hashing). - Reuses per-row group-id buffers across blocks. A new JMH benchmark `BenchmarkGroupingSetsQueriesSSE` exercises the full server→broker flow. ## Benchmark Single-stage engine, 50 segments × 15k rows, low-cardinality dimensions, base aggregation vs. per-row expansion: | Query | Expansion | Base aggregation | Speedup | |---|---|---|---| | `ROLLUP(D1, D2)` | 12.8 ms | 5.3 ms | 2.4x | | `ROLLUP(D1, D2, D3)` | 23.5 ms | 13.7 ms | 1.7x | | `CUBE(D1, D2, D3)` | 35.0 ms | 17.8 ms | 2.0x | | `GROUPING SETS ((D1),(D2),(D3),(D1,D2),())` | 18.6 ms | 8.7 ms | 2.1x | `ROLLUP(D1, D2)` at 5.3 ms is now faster than the plain 3-column GROUP BY (8.8 ms). ## Testing `GroupingSetsQueriesTest` (75 tests) passes, including: - Base-aggregation vs. expansion equivalence across DISTINCTCOUNT, AVG, SUM over CUBE/ROLLUP/GROUPING SETS shapes, with and without null handling (covers the empty-object-intermediate path that exercises the per-set clone). - Coverage for the long-packed generator path. - Existing ROLLUP/CUBE/GROUPING SETS/GROUPING()/GROUPING_ID()/null-handling/MV cases. Pre-commit checks (spotless, license, checkstyle) pass on all touched modules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
