gortiz commented on code in PR #20061:
URL: https://github.com/apache/pinot/pull/20061#discussion_r4549524904
##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/sql/fun/PinotOperatorTable.java:
##########
@@ -141,9 +142,29 @@ public static PinotOperatorTable instance(boolean
nullHandlingEnabled) {
/// [org.apache.pinot.calcite.rel.rules.PinotRuleUtils#isRelocatable]
ignores this flag, so ARRAY_TO_MV filters are
/// still pushed below joins.
public static final PinotSqlFunction ARRAY_TO_MV =
- new PinotSqlFunction("ARRAY_TO_MV", opBinding ->
opBinding.getOperandType(0).getComponentType(),
+ new PinotSqlFunction(ArrayToMvFunctionName.NAME,
+ opBinding -> arrayToMvReturnType(opBinding.getTypeFactory(),
opBinding.getOperandType(0)),
OperandTypes.ARRAY, false);
+ /// The type of `ARRAY_TO_MV(array)`: the component type of `arrayType`. The
type is nullable when the array type or
+ /// the component type is nullable. The MV aggregate gives its MV keys the
same type, so that it can replace a plain
+ /// aggregate over `ARRAY_TO_MV(array)` without changing the row type.
+ ///
+ /// A null array goes into the null group at runtime, as in the single-stage
engine. The type factory applies the
+ /// nullability of an array to the array only, so a nullable array (for
example, from an outer join or a column with
+ /// null handling) usually has a NOT NULL component type. If the key used
that component type, Calcite would remove
+ /// `COALESCE` and fold `IS NULL` on the key, and the null group would give
wrong results.
+ ///
+ /// The type does not depend on the MV aggregate mode, because the `auto`
mode optimizes the already converted plan
+ /// again with the MV aggregate.
+ public static RelDataType arrayToMvReturnType(RelDataTypeFactory
typeFactory, RelDataType arrayType) {
+ RelDataType componentType = arrayType.getComponentType();
+ Preconditions.checkArgument(componentType != null, "ARRAY_TO_MV requires
an array, got: %s",
+ arrayType.getFullTypeString());
+ return typeFactory.createTypeWithNullability(componentType,
+ arrayType.isNullable() || componentType.isNullable());
Review Comment:
Agreed, fixed in 1746b625f4 while keeping the nullable type. A new planner
rule, `PinotArrayToMvNullFilterRule`, moves the null check out of the
expression before the leaf sees it. With `A` for `ARRAY_TO_MV(col)`:
```
pred(COALESCE(A, d)) -> (A IS NOT NULL AND pred(A)) OR (A IS NULL AND
pred(d))
```
The same goes for `CASE WHEN A IS NULL THEN d ELSE A END`. When `d` is a
literal, `pred(d)` is evaluated at planning time, so the filter usually ends up
as `A IS NOT NULL AND pred(A)` or `A IS NULL OR pred(A)`, which the leaf
already runs as plain MV predicates. `pred(A)` keeps the SSE meaning, so `<>`
still means "no element matches". The `IS NOT NULL` part matters without null
handling, where a null row is stored as the default value and `pred(A)` could
match it.
This also fixes a wrong result we had before the type change: the planner
dropped `COALESCE`, so `COALESCE(ARRAY_TO_MV(col), 'x') = 'x'` didn't match the
null rows.
On your example the query returns 2 again. `MvAggregate.json` now has a
`mv_null_filter` section with successful results for `=`, `<>`, `IN`, `<`,
`LIKE`, `NOT`, the `CASE` form, a column default, an empty array, null handling
off, `useMvAggregate=false`, and an MV aggregate above the filter.
`MvAggregatePlans.json` has the plans.
Two limits I left out of scope. `COALESCE` around an `ARRAY_TO_MV` group key
still fails, as on master. A filter that stays in an intermediate stage, for
example above a `LEFT JOIN`, still fails the `ARRAY_TO_MV` validation, the same
as a plain `ARRAY_TO_MV` filter there.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/MultistageGroupByExecutor.java:
##########
@@ -629,4 +677,320 @@ private int[] generateGroupByKeys(DataBlock dataBlock,
int numMatchedRows, Roari
}
return intKeys;
}
+
+ /// Aggregates a block in MV mode. A row creates one group id for each
combination of its MV key elements, so a few
+ /// rows can create a very large number of group ids. When a block needs
more than [#_maxGroupIdsPerBatch] group ids,
+ /// the block is processed in batches of at most that many group ids, and
termination is checked after each batch.
+ private void processMvAggregate(MseBlock.Data block) {
+ if (!processMvBatch(block, null, null)) {
+ processMvAggregateInBatches(block);
+ }
+ }
+
+ /// Splits the rows of `block` into batches of at most
[#_maxGroupIdsPerBatch] group ids and aggregates each batch. A
+ /// row with more combinations than one batch holds is split into ranges of
combinations, and each range goes into its
+ /// own batch. This is correct because a row that adds its values to the
groups of two ranges gives the same result
+ /// as a row that adds them to the groups of both ranges at once.
+ private void processMvAggregateInBatches(MseBlock.Data block) {
+ RowHeapDataBlock rowHeap = block.asRowHeap();
+ DataSchema dataSchema = rowHeap.getDataSchema();
+ AggregationFunction[] aggFunctions = rowHeap.getAggFunctions();
+ List<Object[]> batchRows = new ArrayList<>();
+ IntArrayList batchFroms = new IntArrayList();
+ IntArrayList batchTos = new IntArrayList();
+ int batchSize = 0;
+ for (Object[] row : rowHeap.getRows()) {
+ int numCombinations = countMvCombinations(row);
Review Comment:
Good catch, fixed in 1746b625f4. When `filteredAggregationsSkipEmptyGroups`
is on and every aggregate has a FILTER, the batching path now builds the union
of the matched rows first and only counts and batches those rows. A row that
matches no filter is never looked at, as in the non-batched path.
`testBatchesSkipRowsThatMatchNoFilter` covers it with only filtered aggregates
and a non-matching row of 50,000 x 50,000 combinations, which used to fail with
the overflow error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]