Jackie-Jiang commented on code in PR #20061:
URL: https://github.com/apache/pinot/pull/20061#discussion_r4540803001


##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/sql/fun/PinotOperatorTable.java:
##########
@@ -141,9 +142,29 @@ public static PinotOperatorTable instance(boolean 
nullHandlingEnabled) {
   /// [org.apache.pinot.calcite.rel.rules.PinotRuleUtils#isRelocatable] 
ignores this flag, so ARRAY_TO_MV filters are
   /// still pushed below joins.
   public static final PinotSqlFunction ARRAY_TO_MV =
-      new PinotSqlFunction("ARRAY_TO_MV", opBinding -> 
opBinding.getOperandType(0).getComponentType(),
+      new PinotSqlFunction(ArrayToMvFunctionName.NAME,
+          opBinding -> arrayToMvReturnType(opBinding.getTypeFactory(), 
opBinding.getOperandType(0)),
           OperandTypes.ARRAY, false);
 
+  /// The type of `ARRAY_TO_MV(array)`: the component type of `arrayType`. The 
type is nullable when the array type or
+  /// the component type is nullable. The MV aggregate gives its MV keys the 
same type, so that it can replace a plain
+  /// aggregate over `ARRAY_TO_MV(array)` without changing the row type.
+  ///
+  /// A null array goes into the null group at runtime, as in the single-stage 
engine. The type factory applies the
+  /// nullability of an array to the array only, so a nullable array (for 
example, from an outer join or a column with
+  /// null handling) usually has a NOT NULL component type. If the key used 
that component type, Calcite would remove
+  /// `COALESCE` and fold `IS NULL` on the key, and the null group would give 
wrong results.
+  ///
+  /// The type does not depend on the MV aggregate mode, because the `auto` 
mode optimizes the already converted plan
+  /// again with the MV aggregate.
+  public static RelDataType arrayToMvReturnType(RelDataTypeFactory 
typeFactory, RelDataType arrayType) {
+    RelDataType componentType = arrayType.getComponentType();
+    Preconditions.checkArgument(componentType != null, "ARRAY_TO_MV requires 
an array, got: %s",
+        arrayType.getFullTypeString());
+    return typeFactory.createTypeWithNullability(componentType,
+        arrayType.isNullable() || componentType.isNullable());

Review Comment:
   [P2] Preserve existing nullable-MV COALESCE filters
   
   The corrected nullability preserves COALESCE/CASE, but the existing leaf 
translation strips ARRAY_TO_MV and exposes the raw MV column to 
CaseTransformFunction, which rejects multi-value THEN/ELSE expressions. With a 
nullable MV string column and null handling enabled, `SELECT COUNT(*) FROM tbl 
WHERE COALESCE(ARRAY_TO_MV(strMv), 'x') = 'a'` previously returned the correct 
count 2 for `['a','b']`, NULL, and `['a']`; the updated MvAggregate.json 
fixture now expects an AggregationOperator error. This affects existing queries 
without GROUP BY and persists with `useMvAggregate=false`, since this type 
inference is unconditional. Please retain the corrected nullability while 
adding safe leaf lowering or runtime support for these previously working 
predicates, and keep a successful-result regression test instead of expecting 
an execution error.



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultistageGroupByExecutor.java:
##########
@@ -629,4 +677,320 @@ private int[] generateGroupByKeys(DataBlock dataBlock, 
int numMatchedRows, Roari
     }
     return intKeys;
   }
+
+  /// Aggregates a block in MV mode. A row creates one group id for each 
combination of its MV key elements, so a few
+  /// rows can create a very large number of group ids. When a block needs 
more than [#_maxGroupIdsPerBatch] group ids,
+  /// the block is processed in batches of at most that many group ids, and 
termination is checked after each batch.
+  private void processMvAggregate(MseBlock.Data block) {
+    if (!processMvBatch(block, null, null)) {
+      processMvAggregateInBatches(block);
+    }
+  }
+
+  /// Splits the rows of `block` into batches of at most 
[#_maxGroupIdsPerBatch] group ids and aggregates each batch. A
+  /// row with more combinations than one batch holds is split into ranges of 
combinations, and each range goes into its
+  /// own batch. This is correct because a row that adds its values to the 
groups of two ranges gives the same result
+  /// as a row that adds them to the groups of both ranges at once.
+  private void processMvAggregateInBatches(MseBlock.Data block) {
+    RowHeapDataBlock rowHeap = block.asRowHeap();
+    DataSchema dataSchema = rowHeap.getDataSchema();
+    AggregationFunction[] aggFunctions = rowHeap.getAggFunctions();
+    List<Object[]> batchRows = new ArrayList<>();
+    IntArrayList batchFroms = new IntArrayList();
+    IntArrayList batchTos = new IntArrayList();
+    int batchSize = 0;
+    for (Object[] row : rowHeap.getRows()) {
+      int numCombinations = countMvCombinations(row);

Review Comment:
   [P2] Skip filtered-out rows before counting batch combinations
   
   When `filteredAggregationsSkipEmptyGroups=true` and every aggregate has a 
FILTER, the initial path counts only matched rows, but this fallback counts 
every source row. For example, a matching row with repeated-value MV arrays of 
lengths 1024, 1025, and 1 triggers batching and should give `COUNT(*) FILTER 
(WHERE flag) = 1,049,600` for one distinct group. If another row matches no 
aggregate filter and has three 1,300-element arrays, counting that excluded row 
overflows and throws SERVER_RESOURCE_LIMIT_EXCEEDED. The previous 
implementation never inspected its keys, so the same input returned the valid 
result. Please batch only the union of matched rows when all aggregates are 
filtered and empty groups are skipped. The new batching test always includes an 
unfiltered SUM and therefore misses this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to