yashmayya commented on code in PR #20061:
URL: https://github.com/apache/pinot/pull/20061#discussion_r4593552714
##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotQueryRuleSets.java:
##########
@@ -71,8 +70,8 @@ private PinotQueryRuleSets() {
PinotFilterIntoJoinRule
.instanceWithDescription(PlannerRuleNames.FILTER_INTO_JOIN),
// push filter through an aggregation
- FilterAggregateTransposeRule.Config.DEFAULT
-
.withDescription(PlannerRuleNames.FILTER_AGGREGATE_TRANSPOSE).toRule(),
+ PinotFilterAggregateTransposeRule
Review Comment:
**Suggestion: use the stock `FilterAggregateTransposeRule` in the first
plan, and a small MV rule only in the second plan.**
`PinotFilterAggregateTransposeRule` runs in both plans, so it changes the
plan that old servers run. I found four problems in it:
1. The semi-join blocker (my comment on the rule).
2. `GROUP BY GROUPING SETS ((ARRAY_TO_MV(mcol1), col2), (col2)) HAVING col2
= 'y'` works on master, and master filters `col2` in the leaf. This PR keeps
the filter above the FINAL aggregate, because the rule returns for grouping
sets.
3. `SELECT ARRAY_TO_MV(strMv), w, COUNT(*) FROM (SELECT strMv, ROW_NUMBER()
OVER (ORDER BY k) AS w FROM tbl) GROUP BY ARRAY_TO_MV(strMv), w HAVING w = 1`:
`pushPastProject` puts `ROW_NUMBER() OVER (...) = 1` in a filter below the
window. The query fails at runtime with `Unsupported function: rownumber`.
Master rejects it at planning.
4. The same query with `HAVING ARRAY_TO_MV(strMv) = 'a'`: `canCopyBelow`
goes through the window Project, so the copy changes the row numbers. With the
data of my other comment it returns `(a, 2, 2)`. Correct is `(a, 3, 2)`. No
error.
I tested this rule on top of the PR:
```java
@Override
public void onMatch(RelOptRuleCall call) {
if (!PinotRuleUtils.isMvAggregateEnabled(call) || !readsMvKey(call.rel(0),
call.rel(1))) {
super.onMatch(call);
}
}
```
`readsMvKey` is true when the filter reads a group key that comes from
`ARRAY_TO_MV`. Then the whole filter stays above the aggregate.
Results:
- The rule goes from 108 to 24 code lines. Main code: +50 / −144.
- Cases 1 to 4 give the correct rows or the master plan. The semi-join
queries fail the first plan as on master, and the second plan runs them with
the MV aggregate.
- Of my 41 probe queries that plan on master, 37 get the master plan again.
The other 4 are the lookup-join and `UNNEST` queries that the validation change
rejects.
- The planner tests and the runtime `MvAggregate.json` pass, after I changed
the 10 cases that expect the `HAVING` fix on a leaf-only query.
What is lost:
- The master bug stays for leaf-only queries: `FROM t GROUP BY
ARRAY_TO_MV(tags) HAVING ARRAY_TO_MV(tags) = 'vip'` returns sibling groups. In
round 1 I asked for this fix only where the MV aggregate runs. The fix for
master changes the results and the speed of queries that run today. I think it
is safer in its own PR, with the "copy below" code and its tests.
- A second-plan query gets no leaf pre-filter for a condition on the MV key.
These queries all fail today.
##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotFilterAggregateTransposeRule.java:
##########
@@ -0,0 +1,194 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.calcite.rel.rules;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.rules.FilterAggregateTransposeRule;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUnknownAs;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.sql.fun.SqlBetweenOperator;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Sarg;
+import org.apache.pinot.calcite.rex.PinotSealedSearchOperator;
+
+
+/// Same as [FilterAggregateTransposeRule], except for an aggregate that
groups by `ARRAY_TO_MV(expr)` (see
+/// [ArrayToMvGroupKeys#detect]).
+///
+/// For such an aggregate, a condition on an MV key stays above the aggregate.
Below it, the condition becomes a
+/// multi-value predicate in the leaf stage. That predicate keeps the whole
row when any element matches, so the other
+/// elements of the row also get groups. For example, `GROUP BY
ARRAY_TO_MV(tags) HAVING ARRAY_TO_MV(tags) = 'vip'`
+/// would also return the group of each other tag of a row that has `vip`.
+///
+/// A positive condition on one MV key is also copied below the aggregate when
the `Project` reads directly from a
+/// table scan. A positive condition is `=`, `<`, `<=`, `>`, `>=`, `BETWEEN`,
`IN` or a single range against literals.
+/// The copy keeps only the rows that have a matching element, which are the
only rows that contribute to a matching
+/// group, so the leaf stage aggregates fewer rows. The exact condition stays
above the aggregate. A negated condition
+/// is not copied: as a multi-value predicate, `tags <> 'vip'` and `tags NOT
IN (...)` match a row only when no element
+/// matches. Without a table scan under the `Project`, the copy can end up in
an intermediate stage, where
+/// `ARRAY_TO_MV` is not supported. When a filter under the `Project` already
has an `ARRAY_TO_MV` call, no condition
+/// is copied, so that the rule does not copy the same condition again on its
next match.
+///
+/// The conditions on the other group keys go below the aggregate, into the
`Project` that computes the keys, as with
+/// [FilterAggregateTransposeRule]. The aggregate keeps the `Project` as its
input, so that the MV aggregate rule still
+/// detects the MV keys.
+///
+/// The rule is stateless and thread-safe.
+public class PinotFilterAggregateTransposeRule extends
FilterAggregateTransposeRule {
+
+ public static PinotFilterAggregateTransposeRule
instanceWithDescription(String description) {
+ return new PinotFilterAggregateTransposeRule((Config)
Config.DEFAULT.withDescription(description));
+ }
+
+ private PinotFilterAggregateTransposeRule(Config config) {
+ super(config);
+ }
+
+ @Override
+ public void onMatch(RelOptRuleCall call) {
+ Aggregate aggregate = call.rel(1);
+ List<Integer> mvKeys = ArrayToMvGroupKeys.detect(aggregate);
Review Comment:
**Blocker: a semi-join plus a condition on the MV key returns wrong groups,
with no error.**
Data: `tbl` has `(k=1, strMv=[a, b])`, `(k=2, [b])`, `(k=3, [a, a])`, `(k=5,
[c])`. `ids` has `1, 2, 3`.
```sql
SELECT ARRAY_TO_MV(strMv), COUNT(*) FROM tbl
WHERE k IN (SELECT id FROM ids)
GROUP BY ARRAY_TO_MV(strMv)
HAVING ARRAY_TO_MV(strMv) = 'a'
```
| | Result |
|---|---|
| Correct | `(a, 3)` |
| Merge base | planning error |
| This PR | `(a, 3)`, `(b, 1)` |
The same occurs with an outer filter (`WHERE v = 'b'` returns `(b, 2)`, `(a,
1)`) and with `<>` (returns `(b, 1)`, correct is `(b, 2)`). On an
integration-test cluster (airline data, default options), the same query shape
with `HAVING ARRAY_TO_MV(DivAirports) = 'noodles'` returns 118 groups. Correct
is 1.
Cause: `detect` finds an MV key only in the Project directly below the
aggregate. In the first plan, `PinotProjectJoinTransposeRule` moves
`ARRAY_TO_MV(strMv)` below the semi-join. When this rule runs again, `detect`
returns no key, and `super.onMatch` pushes the condition into the leaf. There
it keeps every element of a row that has a matching element.
The merge base rejects these queries. This is the first plan, so old servers
return the same rows.
Two ways to correct it:
- When `detect` finds nothing, follow each group key through the inputs
(Project, Filter, Join) to find the `ARRAY_TO_MV` call.
- Use the stock rule in the first plan. See my suggestion on
`PinotQueryRuleSets`. These queries then fail the first plan as on master, and
the second plan returns the correct rows.
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java:
##########
@@ -1026,6 +1085,87 @@ public PlannerContext getPlannerContext() {
return _plannerContext;
}
+ /// Builds the dispatchable plan. A plan that fails the `ARRAY_TO_MV`
validation is planned again with the MV
+ /// aggregate (see [#replanWithMvAggregate]).
+ private DispatchableSubPlan toDispatchableSubPlanWithMvAggregateFallback()
{
+ return withMvAggregateFallback(() -> toDispatchableSubPlan(_relRoot,
_plannerContext));
+ }
+
+ /// Runs `planner`, which builds the dispatchable plan from [#_relRoot].
When it fails the `ARRAY_TO_MV` validation
+ /// and [#replanWithMvAggregate] applies, runs it again on the plan with
the MV aggregate. If that also fails, the
+ /// error of the plan with the MV aggregate is thrown, with the first
error as a suppressed exception.
+ private <T> T withMvAggregateFallback(Supplier<T> planner) {
Review Comment:
**Suggestion: decide the fallback inside `compile()`.**
Today `compile()` returns the first plan, and the fallback runs later, in
`planQuery()` or `explain()`. So `CompiledQuery` changes after the broker reads
its table names. That is the cause of the RLS blocker. The broker `explain`
path has no second table-access check, so it can explain a table that the first
plan dropped. I read this in the code and did not run it.
`validateFragments` needs no routing, as
`applyMvAggregateFallbackWithoutImplementation` shows. So `compile()` can make
the decision:
```java
RelRoot relRoot = ...; // first plan
if (mvAggregateFallbackQueryNode != null &&
groupsByArrayToMv(unoptimizedRelRoot.rel)
&& failsArrayToMvValidation(relRoot, plannerContext)) {
plannerContext.close();
// New context: MV aggregate on, nullable ARRAY_TO_MV
plannerContext = getPlannerContext(sqlNodeAndOptions, true);
relRoot = ...; // second plan, from the copy
}
return new CompiledQuery(_envConfig.getDatabase(), sqlQuery, relRoot,
plannerContext, sqlNodeAndOptions);
```
I tested it on top of the PR:
- Main code: +72 / −169. `CompiledQuery` and `PinotDispatchPlanner` are the
master code again. `withMvAggregateFallback`, `replanWithMvAggregate`,
`canFallBackToMvAggregate`, the EXPLAIN pre-check, the tracker
`AtomicReference`, `_unoptimizedRelRoot` and `enableMvAggregateFallback()` go.
`PlannerContext` does not change after construction.
- `getTableNames()` gives the tables of the plan that runs. My end-to-end
RLS query (see my reply to xiangfu0) then returns the filtered result, with
`rlsFiltersApplied=true`.
- The planner tests and `MvAggregate.json` pass, with one changed assertion
in `MvAggregateFallbackTest`.
- The cost is one more `makePlan` call, only for a query that groups by
`ARRAY_TO_MV`.
##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/BaseAggregateOperator.java:
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.runtime.operator;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.pinot.calcite.rel.hint.PinotHintOptions;
+import org.apache.pinot.common.datatable.StatMap;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FunctionContext;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+import
org.apache.pinot.core.query.aggregation.function.AggregationFunctionFactory;
+import
org.apache.pinot.core.query.aggregation.function.AggregationFunctionUtils;
+import
org.apache.pinot.core.query.aggregation.function.CountAggregationFunction;
+import org.apache.pinot.core.util.GroupByUtils;
+import org.apache.pinot.query.parser.CalciteRexExpressionParser;
+import org.apache.pinot.query.planner.logical.RexExpression;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.runtime.blocks.MseBlock;
+import org.apache.pinot.query.runtime.blocks.RowHeapDataBlock;
+import org.apache.pinot.query.runtime.blocks.SuccessMseBlock;
+import org.apache.pinot.query.runtime.plan.OpChainExecutionContext;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.utils.CommonConstants.Server;
+
+
+/// The state and the group-by flow that [AggregateOperator] and
[MvAggregateOperator] share: the aggregation
+/// functions and their filter arguments, the group trim and groups-limit
settings, the stats, the consumption of the
+/// input into a [MultistageGroupByExecutor], and the output block built from
that executor.
+///
+/// A subclass creates the executor in its constructor, and implements
[#consumeInput()] and
+/// [#produceAggregatedBlock()]. The base drives them from [#getNextBlock()]:
it consumes the whole input, builds one
+/// output block, then drops the executor.
+///
+/// Like every [MultiStageOperator], an instance is not thread-safe: one op
chain thread drives it.
+abstract class BaseAggregateOperator extends MultiStageOperator {
Review Comment:
**Suggestion: `MvAggregateOperator extends AggregateOperator`, with no
`BaseAggregateOperator`.**
The two operators differ only in the `mvKeyIndexes` argument of
`MultistageGroupByExecutor`. I tested this on top of the PR:
- `AggregateOperator` is the master code plus a protected constructor that
takes `mvKeyIndexes` (+12 / −5 against master, today +19 / −256).
- `MvAggregateOperator` converts its node to an `AggregateNode` and
overrides only the type, the logger and the explain text.
The three operator files are then +106 / −5 against master. Today they are
+454 / −256. `AggregateOperatorTest`, `MvAggregateOperatorTest`,
`MvKeyExpanderTest`, `ServerPlanRequestVisitorMvAggregateTest` and
`MvAggregate.json` pass with no test change. No code uses `instanceof
AggregateOperator`.
It also makes the stats request of xiangfu0 small. If the subclass keeps
`Type.AGGREGATE`, `MV_AGGREGATE(17)` leaves the PR. The changes in
`MultiStageOperator`, `OperatorTypeDescriptor`, `OperatorTypeRegistry` and
their tests leave it too. I tested that part as well: 8 files, +11 / −57 on top
of the PR, and the tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]