yashmayya commented on code in PR #20061:
URL: https://github.com/apache/pinot/pull/20061#discussion_r4593552714


##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotQueryRuleSets.java:
##########
@@ -71,8 +70,8 @@ private PinotQueryRuleSets() {
       PinotFilterIntoJoinRule
           .instanceWithDescription(PlannerRuleNames.FILTER_INTO_JOIN),
       // push filter through an aggregation
-      FilterAggregateTransposeRule.Config.DEFAULT
-          
.withDescription(PlannerRuleNames.FILTER_AGGREGATE_TRANSPOSE).toRule(),
+      PinotFilterAggregateTransposeRule

Review Comment:
   **Suggestion: use the stock `FilterAggregateTransposeRule` in the first 
plan, and a small MV rule only in the second plan.**
   
   `PinotFilterAggregateTransposeRule` runs in both plans, so it changes the 
plan that old servers run. I found four problems in it:
   
   1. The semi-join blocker (my comment on the rule).
   2. `GROUP BY GROUPING SETS ((ARRAY_TO_MV(mcol1), col2), (col2)) HAVING col2 
= 'y'` works on master, and master filters `col2` in the leaf. This PR keeps 
the filter above the FINAL aggregate, because the rule returns for grouping 
sets.
   3. `SELECT ARRAY_TO_MV(strMv), w, COUNT(*) FROM (SELECT strMv, ROW_NUMBER() 
OVER (ORDER BY k) AS w FROM tbl) GROUP BY ARRAY_TO_MV(strMv), w HAVING w = 1`: 
`pushPastProject` puts `ROW_NUMBER() OVER (...) = 1` in a filter below the 
window. The query fails at runtime with `Unsupported function: rownumber`. 
Master rejects it at planning.
   4. The same query with `HAVING ARRAY_TO_MV(strMv) = 'a'`: `canCopyBelow` 
goes through the window Project, so the copy changes the row numbers. With the 
data of my other comment it returns `(a, 2, 2)`. Correct is `(a, 3, 2)`. No 
error.
   
   I tested this rule on top of the PR:
   
   ```java
   @Override
   public void onMatch(RelOptRuleCall call) {
     if (!PinotRuleUtils.isMvAggregateEnabled(call) || !readsMvKey(call.rel(0), 
call.rel(1))) {
       super.onMatch(call);
     }
   }
   ```
   
   `readsMvKey` is true when the filter reads a group key that comes from 
`ARRAY_TO_MV`. Then the whole filter stays above the aggregate.
   
   Results:
   - The rule goes from 108 to 24 code lines. Main code: +50 / −144.
   - Cases 1 to 4 give the correct rows or the master plan. The semi-join 
queries fail the first plan as on master, and the second plan runs them with 
the MV aggregate.
   - Of my 41 probe queries that plan on master, 37 get the master plan again. 
The other 4 are the lookup-join and `UNNEST` queries that the validation change 
rejects.
   - The planner tests and the runtime `MvAggregate.json` pass, after I changed 
the 10 cases that expect the `HAVING` fix on a leaf-only query.
   
   What is lost:
   - The master bug stays for leaf-only queries: `FROM t GROUP BY 
ARRAY_TO_MV(tags) HAVING ARRAY_TO_MV(tags) = 'vip'` returns sibling groups. In 
round 1 I asked for this fix only where the MV aggregate runs. The fix for 
master changes the results and the speed of queries that run today. I think it 
is safer in its own PR, with the "copy below" code and its tests.
   - A second-plan query gets no leaf pre-filter for a condition on the MV key. 
These queries all fail today.



##########
pinot-query-planner/src/main/java/org/apache/pinot/calcite/rel/rules/PinotFilterAggregateTransposeRule.java:
##########
@@ -0,0 +1,194 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.calcite.rel.rules;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Aggregate;
+import org.apache.calcite.rel.core.Filter;
+import org.apache.calcite.rel.core.Project;
+import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.rules.FilterAggregateTransposeRule;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexUnknownAs;
+import org.apache.calcite.rex.RexUtil;
+import org.apache.calcite.sql.fun.SqlBetweenOperator;
+import org.apache.calcite.tools.RelBuilder;
+import org.apache.calcite.util.ImmutableBitSet;
+import org.apache.calcite.util.Sarg;
+import org.apache.pinot.calcite.rex.PinotSealedSearchOperator;
+
+
+/// Same as [FilterAggregateTransposeRule], except for an aggregate that 
groups by `ARRAY_TO_MV(expr)` (see
+/// [ArrayToMvGroupKeys#detect]).
+///
+/// For such an aggregate, a condition on an MV key stays above the aggregate. 
Below it, the condition becomes a
+/// multi-value predicate in the leaf stage. That predicate keeps the whole 
row when any element matches, so the other
+/// elements of the row also get groups. For example, `GROUP BY 
ARRAY_TO_MV(tags) HAVING ARRAY_TO_MV(tags) = 'vip'`
+/// would also return the group of each other tag of a row that has `vip`.
+///
+/// A positive condition on one MV key is also copied below the aggregate when 
the `Project` reads directly from a
+/// table scan. A positive condition is `=`, `<`, `<=`, `>`, `>=`, `BETWEEN`, 
`IN` or a single range against literals.
+/// The copy keeps only the rows that have a matching element, which are the 
only rows that contribute to a matching
+/// group, so the leaf stage aggregates fewer rows. The exact condition stays 
above the aggregate. A negated condition
+/// is not copied: as a multi-value predicate, `tags <> 'vip'` and `tags NOT 
IN (...)` match a row only when no element
+/// matches. Without a table scan under the `Project`, the copy can end up in 
an intermediate stage, where
+/// `ARRAY_TO_MV` is not supported. When a filter under the `Project` already 
has an `ARRAY_TO_MV` call, no condition
+/// is copied, so that the rule does not copy the same condition again on its 
next match.
+///
+/// The conditions on the other group keys go below the aggregate, into the 
`Project` that computes the keys, as with
+/// [FilterAggregateTransposeRule]. The aggregate keeps the `Project` as its 
input, so that the MV aggregate rule still
+/// detects the MV keys.
+///
+/// The rule is stateless and thread-safe.
+public class PinotFilterAggregateTransposeRule extends 
FilterAggregateTransposeRule {
+
+  public static PinotFilterAggregateTransposeRule 
instanceWithDescription(String description) {
+    return new PinotFilterAggregateTransposeRule((Config) 
Config.DEFAULT.withDescription(description));
+  }
+
+  private PinotFilterAggregateTransposeRule(Config config) {
+    super(config);
+  }
+
+  @Override
+  public void onMatch(RelOptRuleCall call) {
+    Aggregate aggregate = call.rel(1);
+    List<Integer> mvKeys = ArrayToMvGroupKeys.detect(aggregate);

Review Comment:
   **Blocker: a semi-join plus a condition on the MV key returns wrong groups, 
with no error.**
   
   Data: `tbl` has `(k=1, strMv=[a, b])`, `(k=2, [b])`, `(k=3, [a, a])`, `(k=5, 
[c])`. `ids` has `1, 2, 3`.
   
   ```sql
   SELECT ARRAY_TO_MV(strMv), COUNT(*) FROM tbl
   WHERE k IN (SELECT id FROM ids)
   GROUP BY ARRAY_TO_MV(strMv)
   HAVING ARRAY_TO_MV(strMv) = 'a'
   ```
   
   | | Result |
   |---|---|
   | Correct | `(a, 3)` |
   | Merge base | planning error |
   | This PR | `(a, 3)`, `(b, 1)` |
   
   The same occurs with an outer filter (`WHERE v = 'b'` returns `(b, 2)`, `(a, 
1)`) and with `<>` (returns `(b, 1)`, correct is `(b, 2)`). On an 
integration-test cluster (airline data, default options), the same query shape 
with `HAVING ARRAY_TO_MV(DivAirports) = 'noodles'` returns 118 groups. Correct 
is 1.
   
   Cause: `detect` finds an MV key only in the Project directly below the 
aggregate. In the first plan, `PinotProjectJoinTransposeRule` moves 
`ARRAY_TO_MV(strMv)` below the semi-join. When this rule runs again, `detect` 
returns no key, and `super.onMatch` pushes the condition into the leaf. There 
it keeps every element of a row that has a matching element.
   
   The merge base rejects these queries. This is the first plan, so old servers 
return the same rows.
   
   Two ways to correct it:
   - When `detect` finds nothing, follow each group key through the inputs 
(Project, Filter, Join) to find the `ARRAY_TO_MV` call.
   - Use the stock rule in the first plan. See my suggestion on 
`PinotQueryRuleSets`. These queries then fail the first plan as on master, and 
the second plan returns the correct rows.



##########
pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java:
##########
@@ -1026,6 +1085,87 @@ public PlannerContext getPlannerContext() {
       return _plannerContext;
     }
 
+    /// Builds the dispatchable plan. A plan that fails the `ARRAY_TO_MV` 
validation is planned again with the MV
+    /// aggregate (see [#replanWithMvAggregate]).
+    private DispatchableSubPlan toDispatchableSubPlanWithMvAggregateFallback() 
{
+      return withMvAggregateFallback(() -> toDispatchableSubPlan(_relRoot, 
_plannerContext));
+    }
+
+    /// Runs `planner`, which builds the dispatchable plan from [#_relRoot]. 
When it fails the `ARRAY_TO_MV` validation
+    /// and [#replanWithMvAggregate] applies, runs it again on the plan with 
the MV aggregate. If that also fails, the
+    /// error of the plan with the MV aggregate is thrown, with the first 
error as a suppressed exception.
+    private <T> T withMvAggregateFallback(Supplier<T> planner) {

Review Comment:
   **Suggestion: decide the fallback inside `compile()`.**
   
   Today `compile()` returns the first plan, and the fallback runs later, in 
`planQuery()` or `explain()`. So `CompiledQuery` changes after the broker reads 
its table names. That is the cause of the RLS blocker. The broker `explain` 
path has no second table-access check, so it can explain a table that the first 
plan dropped. I read this in the code and did not run it.
   
   `validateFragments` needs no routing, as 
`applyMvAggregateFallbackWithoutImplementation` shows. So `compile()` can make 
the decision:
   
   ```java
   RelRoot relRoot = ...;  // first plan
   if (mvAggregateFallbackQueryNode != null && 
groupsByArrayToMv(unoptimizedRelRoot.rel)
       && failsArrayToMvValidation(relRoot, plannerContext)) {
     plannerContext.close();
     // New context: MV aggregate on, nullable ARRAY_TO_MV
     plannerContext = getPlannerContext(sqlNodeAndOptions, true);
     relRoot = ...;  // second plan, from the copy
   }
   return new CompiledQuery(_envConfig.getDatabase(), sqlQuery, relRoot, 
plannerContext, sqlNodeAndOptions);
   ```
   
   I tested it on top of the PR:
   - Main code: +72 / −169. `CompiledQuery` and `PinotDispatchPlanner` are the 
master code again. `withMvAggregateFallback`, `replanWithMvAggregate`, 
`canFallBackToMvAggregate`, the EXPLAIN pre-check, the tracker 
`AtomicReference`, `_unoptimizedRelRoot` and `enableMvAggregateFallback()` go. 
`PlannerContext` does not change after construction.
   - `getTableNames()` gives the tables of the plan that runs. My end-to-end 
RLS query (see my reply to xiangfu0) then returns the filtered result, with 
`rlsFiltersApplied=true`.
   - The planner tests and `MvAggregate.json` pass, with one changed assertion 
in `MvAggregateFallbackTest`.
   - The cost is one more `makePlan` call, only for a query that groups by 
`ARRAY_TO_MV`.



##########
pinot-query-runtime/src/main/java/org/apache/pinot/query/runtime/operator/BaseAggregateOperator.java:
##########
@@ -0,0 +1,325 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.query.runtime.operator;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.calcite.rel.RelFieldCollation;
+import org.apache.pinot.calcite.rel.hint.PinotHintOptions;
+import org.apache.pinot.common.datatable.StatMap;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FunctionContext;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.common.utils.config.QueryOptionsUtils;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+import 
org.apache.pinot.core.query.aggregation.function.AggregationFunctionFactory;
+import 
org.apache.pinot.core.query.aggregation.function.AggregationFunctionUtils;
+import 
org.apache.pinot.core.query.aggregation.function.CountAggregationFunction;
+import org.apache.pinot.core.util.GroupByUtils;
+import org.apache.pinot.query.parser.CalciteRexExpressionParser;
+import org.apache.pinot.query.planner.logical.RexExpression;
+import org.apache.pinot.query.planner.plannode.PlanNode;
+import org.apache.pinot.query.runtime.blocks.MseBlock;
+import org.apache.pinot.query.runtime.blocks.RowHeapDataBlock;
+import org.apache.pinot.query.runtime.blocks.SuccessMseBlock;
+import org.apache.pinot.query.runtime.plan.OpChainExecutionContext;
+import org.apache.pinot.spi.exception.QueryErrorCode;
+import org.apache.pinot.spi.utils.CommonConstants.Server;
+
+
+/// The state and the group-by flow that [AggregateOperator] and 
[MvAggregateOperator] share: the aggregation
+/// functions and their filter arguments, the group trim and groups-limit 
settings, the stats, the consumption of the
+/// input into a [MultistageGroupByExecutor], and the output block built from 
that executor.
+///
+/// A subclass creates the executor in its constructor, and implements 
[#consumeInput()] and
+/// [#produceAggregatedBlock()]. The base drives them from [#getNextBlock()]: 
it consumes the whole input, builds one
+/// output block, then drops the executor.
+///
+/// Like every [MultiStageOperator], an instance is not thread-safe: one op 
chain thread drives it.
+abstract class BaseAggregateOperator extends MultiStageOperator {

Review Comment:
   **Suggestion: `MvAggregateOperator extends AggregateOperator`, with no 
`BaseAggregateOperator`.**
   
   The two operators differ only in the `mvKeyIndexes` argument of 
`MultistageGroupByExecutor`. I tested this on top of the PR:
   
   - `AggregateOperator` is the master code plus a protected constructor that 
takes `mvKeyIndexes` (+12 / −5 against master, today +19 / −256).
   - `MvAggregateOperator` converts its node to an `AggregateNode` and 
overrides only the type, the logger and the explain text.
   
   The three operator files are then +106 / −5 against master. Today they are 
+454 / −256. `AggregateOperatorTest`, `MvAggregateOperatorTest`, 
`MvKeyExpanderTest`, `ServerPlanRequestVisitorMvAggregateTest` and 
`MvAggregate.json` pass with no test change. No code uses `instanceof 
AggregateOperator`.
   
   It also makes the stats request of xiangfu0 small. If the subclass keeps 
`Type.AGGREGATE`, `MV_AGGREGATE(17)` leaves the PR. The changes in 
`MultiStageOperator`, `OperatorTypeDescriptor`, `OperatorTypeRegistry` and 
their tests leave it too. I tested that part as well: 8 files, +11 / −57 on top 
of the PR, and the tests pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to