xiangfu0 commented on code in PR #19264:
URL: https://github.com/apache/pinot/pull/19264#discussion_r4148021602
##########
pinot-core/src/main/java/org/apache/pinot/core/util/GroupByUtils.java:
##########
@@ -87,6 +98,125 @@ public static GroupByResultsBlock
buildGroupingSetsResultsBlock(QueryContext que
return resultsBlock;
}
+ /// Builds the segment-level [GroupByResultsBlock] for a grouping-set query
from BASE groups, i.e. groups
+ /// aggregated once over the union of all grouping-set columns (a plain
GROUP BY), rather than by expanding
+ /// every input row into one group per grouping set. Each base group is then
projected into each grouping set:
+ /// its rolled-up (non-participating) columns are set to `null`, the
`$groupingId` discriminator is appended,
+ /// and the base group's aggregation intermediates are merged into the
derived group. This moves the per-set
+ /// fan-out from O(rows) to O(base groups), reusing the fast plain-GROUP-BY
scan path.
+ ///
+ /// Because a base group's intermediate result flows into every grouping
set, and [AggregationFunction#merge]
+ /// mutates its first argument, each base intermediate is cloned per set
before it becomes a merge target (see
+ /// [#cloneIntermediate]). This keeps the derivation exact for object-backed
accumulators (AVG, DISTINCTCOUNT,
+ /// percentiles, ...) as well as scalar ones.
+ ///
+ /// @param discriminatorColumnIndex index of the synthetic $groupingId
column, i.e. the number of union
+ /// group-by columns
+ public static GroupByResultsBlock
buildGroupingSetsResultsBlockFromBaseGroups(QueryContext queryContext,
+ DataSchema dataSchema, AggregationGroupByResult baseResult, int
discriminatorColumnIndex,
+ boolean numGroupsLimitReached, boolean numGroupsWarningLimitReached) {
+ AggregationFunction[] aggregationFunctions =
queryContext.getAggregationFunctions();
+ assert aggregationFunctions != null;
+ int numAggregationFunctions = aggregationFunctions.length;
+ List<int[]> groupingSets = queryContext.getGroupingSets();
+ int numSets = groupingSets.size();
+ int numUnionColumns = discriminatorColumnIndex;
+ // Per grouping set: membership mask over the union columns (true =
participates, false = rolled up to NULL).
+ boolean[][] setContains = new boolean[numSets][numUnionColumns];
+ for (int s = 0; s < numSets; s++) {
+ for (int columnIndex : groupingSets.get(s)) {
+ setContains[s][columnIndex] = true;
+ }
+ }
+
+ // Derived group table keyed on (projected union values..., $groupingId).
Values layout mirrors the record
+ // schema: key columns first, then the aggregation intermediates.
+ Map<Key, Record> derived = new HashMap<>();
Review Comment:
Fixed in a9b179e. When base groups × sets can exceed max(numGroupsLimit,
numSets), derivation now processes sets coarsest-first into one capped map
instead of materializing every per-set map first. It continues merging into
admitted groups after the cap fills, and the mixed BASE/FULL merge cannot grow
past the same cap. Dropped groups set groupsTrimmed and numGroupsLimitReached;
the retained derived count also participates in the warning-limit flag. Focused
derive/mixed-layout/streaming tests passed 9/9 locally (with the local compiler
classpath workaround documented in the PR body); exact-head CI is pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
