Jackie-Jiang commented on code in PR #20061:
URL: https://github.com/apache/pinot/pull/20061#discussion_r4540803001
##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/sql/fun/PinotOperatorTable.java:
##########
@@ -141,9 +142,29 @@ public static PinotOperatorTable instance(boolean
nullHandlingEnabled) {
/// [org.apache.pinot.calcite.rel.rules.PinotRuleUtils#isRelocatable]
ignores this flag, so ARRAY_TO_MV filters are
/// still pushed below joins.
public static final PinotSqlFunction ARRAY_TO_MV =
- new PinotSqlFunction("ARRAY_TO_MV", opBinding ->
opBinding.getOperandType(0).getComponentType(),
+ new PinotSqlFunction(ArrayToMvFunctionName.NAME,
+ opBinding -> arrayToMvReturnType(opBinding.getTypeFactory(),
opBinding.getOperandType(0)),
OperandTypes.ARRAY, false);
+ /// The type of `ARRAY_TO_MV(array)`: the component type of `arrayType`. The
type is nullable when the array type or
+ /// the component type is nullable. The MV aggregate gives its MV keys the
same type, so that it can replace a plain
+ /// aggregate over `ARRAY_TO_MV(array)` without changing the row type.
+ ///
+ /// A null array goes into the null group at runtime, as in the single-stage
engine. The type factory applies the
+ /// nullability of an array to the array only, so a nullable array (for
example, from an outer join or a column with
+ /// null handling) usually has a NOT NULL component type. If the key used
that component type, Calcite would remove
+ /// `COALESCE` and fold `IS NULL` on the key, and the null group would give
wrong results.
+ ///
+ /// The type does not depend on the MV aggregate mode, because the `auto`
mode optimizes the already converted plan
+ /// again with the MV aggregate.
+ public static RelDataType arrayToMvReturnType(RelDataTypeFactory
typeFactory, RelDataType arrayType) {
+ RelDataType componentType = arrayType.getComponentType();
+ Preconditions.checkArgument(componentType != null, "ARRAY_TO_MV requires
an array, got: %s",
+ arrayType.getFullTypeString());
+ return typeFactory.createTypeWithNullability(componentType,
+ arrayType.isNullable() || componentType.isNullable());
Review Comment:
[P2] Preserve existing nullable-MV COALESCE filters
The corrected nullability preserves COALESCE/CASE, but the existing leaf
translation strips ARRAY_TO_MV and exposes the raw MV column to
CaseTransformFunction, which rejects multi-value THEN/ELSE expressions. With a
nullable MV string column and null handling enabled, `SELECT COUNT(*) FROM tbl
WHERE COALESCE(ARRAY_TO_MV(strMv), 'x') = 'a'` previously returned the correct
count 2 for `['a','b']`, NULL, and `['a']`; the updated MvAggregate.json
fixture now expects an AggregationOperator error. This affects existing queries
without GROUP BY and persists with `useMvAggregate=false`, since this type
inference is unconditional. Please retain the corrected nullability while
adding safe leaf lowering or runtime support for these previously working
predicates, and keep a successful-result regression test instead of expecting
an execution error.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultistageGroupByExecutor.java:
##########
@@ -629,4 +677,320 @@ private int[] generateGroupByKeys(DataBlock dataBlock,
int numMatchedRows, Roari
}
return intKeys;
}
+
+ /// Aggregates a block in MV mode. A row creates one group id for each
combination of its MV key elements, so a few
+ /// rows can create a very large number of group ids. When a block needs
more than [#_maxGroupIdsPerBatch] group ids,
+ /// the block is processed in batches of at most that many group ids, and
termination is checked after each batch.
+ private void processMvAggregate(MseBlock.Data block) {
+ if (!processMvBatch(block, null, null)) {
+ processMvAggregateInBatches(block);
+ }
+ }
+
+ /// Splits the rows of `block` into batches of at most
[#_maxGroupIdsPerBatch] group ids and aggregates each batch. A
+ /// row with more combinations than one batch holds is split into ranges of
combinations, and each range goes into its
+ /// own batch. This is correct because a row that adds its values to the
groups of two ranges gives the same result
+ /// as a row that adds them to the groups of both ranges at once.
+ private void processMvAggregateInBatches(MseBlock.Data block) {
+ RowHeapDataBlock rowHeap = block.asRowHeap();
+ DataSchema dataSchema = rowHeap.getDataSchema();
+ AggregationFunction[] aggFunctions = rowHeap.getAggFunctions();
+ List<Object[]> batchRows = new ArrayList<>();
+ IntArrayList batchFroms = new IntArrayList();
+ IntArrayList batchTos = new IntArrayList();
+ int batchSize = 0;
+ for (Object[] row : rowHeap.getRows()) {
+ int numCombinations = countMvCombinations(row);
Review Comment:
[P2] Skip filtered-out rows before counting batch combinations
When `filteredAggregationsSkipEmptyGroups=true` and every aggregate has a
FILTER, the initial path counts only matched rows, but this fallback counts
every source row. For example, a matching row with repeated-value MV arrays of
lengths 1024, 1025, and 1 triggers batching and should give `COUNT(*) FILTER
(WHERE flag) = 1,049,600` for one distinct group. If another row matches no
aggregate filter and has three 1,300-element arrays, counting that excluded row
overflows and throws SERVER_RESOURCE_LIMIT_EXCEEDED. The previous
implementation never inspected its keys, so the same input returned the valid
result. Please batch only the union of matched rows when all aggregates are
filtered and empty groups are skipped. The new batching test always includes an
unfiltered SUM and therefore misses this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]