xiangfu0 commented on PR #19264: URL: https://github.com/apache/pinot/pull/19264#issuecomment-5354389442
Made grouping-set trimming **configurable per stage** so accuracy vs. memory/latency can be tuned (pushed 20fa894). **Where trimming happens now (base aggregation):** | Stage | Cap | Notes | |---|---|---| | Segment scan | base groups ≤ `numGroupsLimit` | same memory guardrail as a plain GROUP BY | | Server combine — base merge | base groups ≤ `numGroupsLimit` | trim-disabled table | | Server combine — **derive** | **configurable** (new) | per-set top-K via `groupingSetsServerTrimSize`, default off (keep all) | | Broker reduce | ORDER BY + LIMIT | the exact, final trim | **New knob:** `groupingSetsServerTrimSize` — for a base-aggregation grouping-set query **with ORDER BY**, keep at most K groups **within each grouping set** on the server after the derive (bucketed by `$groupingId`), so a global top-K can never starve a low-magnitude set such as the grand total. This bounds each server's derived output for high-cardinality unions, at the cost of an approximate per-server top-K (the broker still applies the exact final ORDER BY + LIMIT). - **Default (unset / ≤ 0):** keep all derived groups and let only the broker trim — exact, but higher memory/network. - **Set to K:** bounded per-set server output — cheaper, approximate. - Ignored without ORDER BY. The per-set bucketing (reusing the same anti-starvation logic as the legacy expansion path's segment trim) is the key: it never drops an entire grouping set, only ranks within each. Added an integration test asserting a large K is exact (no group dropped) and a small K (K=1) bounds the output while every set — including the grand total — still survives. All 76 grouping-sets tests pass; checkstyle/license/spotless clean. This gives operators the accuracy/cost dial you asked for; we can add segment-stage or broker-stage variants later on the same pattern if a use case wants them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
