gortiz commented on code in PR #20061:
URL: https://github.com/apache/pinot/pull/20061#discussion_r4549524904
##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/sql/fun/PinotOperatorTable.java:
##########
@@ -141,9 +142,29 @@ public static PinotOperatorTable instance(boolean
nullHandlingEnabled) {
/// [org.apache.pinot.calcite.rel.rules.PinotRuleUtils#isRelocatable]
ignores this flag, so ARRAY_TO_MV filters are
/// still pushed below joins.
public static final PinotSqlFunction ARRAY_TO_MV =
- new PinotSqlFunction("ARRAY_TO_MV", opBinding ->
opBinding.getOperandType(0).getComponentType(),
+ new PinotSqlFunction(ArrayToMvFunctionName.NAME,
+ opBinding -> arrayToMvReturnType(opBinding.getTypeFactory(),
opBinding.getOperandType(0)),
OperandTypes.ARRAY, false);
+ /// The type of `ARRAY_TO_MV(array)`: the component type of `arrayType`. The
type is nullable when the array type or
+ /// the component type is nullable. The MV aggregate gives its MV keys the
same type, so that it can replace a plain
+ /// aggregate over `ARRAY_TO_MV(array)` without changing the row type.
+ ///
+ /// A null array goes into the null group at runtime, as in the single-stage
engine. The type factory applies the
+ /// nullability of an array to the array only, so a nullable array (for
example, from an outer join or a column with
+ /// null handling) usually has a NOT NULL component type. If the key used
that component type, Calcite would remove
+ /// `COALESCE` and fold `IS NULL` on the key, and the null group would give
wrong results.
+ ///
+ /// The type does not depend on the MV aggregate mode, because the `auto`
mode optimizes the already converted plan
+ /// again with the MV aggregate.
+ public static RelDataType arrayToMvReturnType(RelDataTypeFactory
typeFactory, RelDataType arrayType) {
+ RelDataType componentType = arrayType.getComponentType();
+ Preconditions.checkArgument(componentType != null, "ARRAY_TO_MV requires
an array, got: %s",
+ arrayType.getFullTypeString());
+ return typeFactory.createTypeWithNullability(componentType,
+ arrayType.isNullable() || componentType.isNullable());
Review Comment:
Agreed, fixed in 1746b625f4 while keeping the nullable type. A new planner
rule, `PinotArrayToMvCoalesceFilterRule`, moves the null check out of the
expression before the leaf sees it. With `A` for `ARRAY_TO_MV(col)`:
```
pred(COALESCE(A, d)) -> (A IS NOT NULL AND pred(A)) OR (A IS NULL AND
pred(d))
```
The same goes for `CASE WHEN A IS NULL THEN d ELSE A END`. When `d` is a
literal, `pred(d)` is evaluated at planning time, so the filter usually ends up
as `A IS NOT NULL AND pred(A)` or `A IS NULL OR pred(A)`, which the leaf
already runs as plain MV predicates. `pred(A)` keeps the SSE meaning, so `<>`
still means "no element matches". The `IS NOT NULL` part matters without null
handling, where a null row is stored as the default value and `pred(A)` could
match it.
This also fixes a wrong result we had before the type change: the planner
dropped `COALESCE`, so `COALESCE(ARRAY_TO_MV(col), 'x') = 'x'` didn't match the
null rows.
On your example the query returns 2 again. `MvAggregate.json` now has a
`mv_null_filter` section with successful results for `=`, `<>`, `IN`, `<`,
`LIKE`, `NOT`, the `CASE` form, a column default, an empty array, null handling
off, `useMvAggregate=false`, and an MV aggregate above the filter.
`MvAggregatePlans.json` has the plans.
Two limits I left out of scope. `COALESCE` around an `ARRAY_TO_MV` group key
still fails, as on master. A filter that stays in an intermediate stage, for
example above a `LEFT JOIN`, still fails the `ARRAY_TO_MV` validation, the same
as a plain `ARRAY_TO_MV` filter there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]