gortiz commented on code in PR #20061:
URL: https://github.com/apache/pinot/pull/20061#discussion_r4549524904


##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/sql/fun/PinotOperatorTable.java:
##########
@@ -141,9 +142,29 @@ public static PinotOperatorTable instance(boolean 
nullHandlingEnabled) {
   /// [org.apache.pinot.calcite.rel.rules.PinotRuleUtils#isRelocatable] 
ignores this flag, so ARRAY_TO_MV filters are
   /// still pushed below joins.
   public static final PinotSqlFunction ARRAY_TO_MV =
-      new PinotSqlFunction("ARRAY_TO_MV", opBinding -> 
opBinding.getOperandType(0).getComponentType(),
+      new PinotSqlFunction(ArrayToMvFunctionName.NAME,
+          opBinding -> arrayToMvReturnType(opBinding.getTypeFactory(), 
opBinding.getOperandType(0)),
           OperandTypes.ARRAY, false);
 
+  /// The type of `ARRAY_TO_MV(array)`: the component type of `arrayType`. The 
type is nullable when the array type or
+  /// the component type is nullable. The MV aggregate gives its MV keys the 
same type, so that it can replace a plain
+  /// aggregate over `ARRAY_TO_MV(array)` without changing the row type.
+  ///
+  /// A null array goes into the null group at runtime, as in the single-stage 
engine. The type factory applies the
+  /// nullability of an array to the array only, so a nullable array (for 
example, from an outer join or a column with
+  /// null handling) usually has a NOT NULL component type. If the key used 
that component type, Calcite would remove
+  /// `COALESCE` and fold `IS NULL` on the key, and the null group would give 
wrong results.
+  ///
+  /// The type does not depend on the MV aggregate mode, because the `auto` 
mode optimizes the already converted plan
+  /// again with the MV aggregate.
+  public static RelDataType arrayToMvReturnType(RelDataTypeFactory 
typeFactory, RelDataType arrayType) {
+    RelDataType componentType = arrayType.getComponentType();
+    Preconditions.checkArgument(componentType != null, "ARRAY_TO_MV requires 
an array, got: %s",
+        arrayType.getFullTypeString());
+    return typeFactory.createTypeWithNullability(componentType,
+        arrayType.isNullable() || componentType.isNullable());

Review Comment:
   Agreed, fixed in 1746b625f4 while keeping the nullable type. A new planner 
rule, `PinotArrayToMvNullFilterRule`, moves the null check out of the 
expression before the leaf sees it. With `A` for `ARRAY_TO_MV(col)`:
   
   ```
   pred(COALESCE(A, d))  ->  (A IS NOT NULL AND pred(A)) OR (A IS NULL AND 
pred(d))
   ```
   
   The same goes for `CASE WHEN A IS NULL THEN d ELSE A END`. When `d` is a 
literal, `pred(d)` is evaluated at planning time, so the filter usually ends up 
as `A IS NOT NULL AND pred(A)` or `A IS NULL OR pred(A)`, which the leaf 
already runs as plain MV predicates. `pred(A)` keeps the SSE meaning, so `<>` 
still means "no element matches". The `IS NOT NULL` part matters without null 
handling, where a null row is stored as the default value and `pred(A)` could 
match it.
   
   This also fixes a wrong result we had before the type change: the planner 
dropped `COALESCE`, so `COALESCE(ARRAY_TO_MV(col), 'x') = 'x'` didn't match the 
null rows.
   
   On your example the query returns 2 again. `MvAggregate.json` now has a 
`mv_null_filter` section with successful results for `=`, `<>`, `IN`, `<`, 
`LIKE`, `NOT`, the `CASE` form, a column default, an empty array, null handling 
off, `useMvAggregate=false`, and an MV aggregate above the filter. 
`MvAggregatePlans.json` has the plans.
   
   Two limits I left out of scope. `COALESCE` around an `ARRAY_TO_MV` group key 
still fails, as on master. A filter that stays in an intermediate stage, for 
example above a `LEFT JOIN`, still fails the `ARRAY_TO_MV` validation, the same 
as a plain `ARRAY_TO_MV` filter there.
   



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultistageGroupByExecutor.java:
##########
@@ -629,4 +677,320 @@ private int[] generateGroupByKeys(DataBlock dataBlock, 
int numMatchedRows, Roari
     }
     return intKeys;
   }
+
+  /// Aggregates a block in MV mode. A row creates one group id for each 
combination of its MV key elements, so a few
+  /// rows can create a very large number of group ids. When a block needs 
more than [#_maxGroupIdsPerBatch] group ids,
+  /// the block is processed in batches of at most that many group ids, and 
termination is checked after each batch.
+  private void processMvAggregate(MseBlock.Data block) {
+    if (!processMvBatch(block, null, null)) {
+      processMvAggregateInBatches(block);
+    }
+  }
+
+  /// Splits the rows of `block` into batches of at most 
[#_maxGroupIdsPerBatch] group ids and aggregates each batch. A
+  /// row with more combinations than one batch holds is split into ranges of 
combinations, and each range goes into its
+  /// own batch. This is correct because a row that adds its values to the 
groups of two ranges gives the same result
+  /// as a row that adds them to the groups of both ranges at once.
+  private void processMvAggregateInBatches(MseBlock.Data block) {
+    RowHeapDataBlock rowHeap = block.asRowHeap();
+    DataSchema dataSchema = rowHeap.getDataSchema();
+    AggregationFunction[] aggFunctions = rowHeap.getAggFunctions();
+    List<Object[]> batchRows = new ArrayList<>();
+    IntArrayList batchFroms = new IntArrayList();
+    IntArrayList batchTos = new IntArrayList();
+    int batchSize = 0;
+    for (Object[] row : rowHeap.getRows()) {
+      int numCombinations = countMvCombinations(row);

Review Comment:
   Good catch, fixed in 1746b625f4. When `filteredAggregationsSkipEmptyGroups` 
is on and every aggregate has a FILTER, the batching path now builds the union 
of the matched rows first and only counts and batches those rows. A row that 
matches no filter is never looked at, as in the non-batched path. 
`testBatchesSkipRowsThatMatchNoFilter` covers it with only filtered aggregates 
and a non-matching row of 50,000 x 50,000 combinations, which used to fail with 
the overflow error.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to