yashmayya commented on code in PR #20061:
URL: https://github.com/apache/pinot/pull/20061#discussion_r4593551520


##########
pinot-query-planner/src/main/java/org/apache/pinot/query/QueryEnvironment.java:
##########
@@ -1026,6 +1085,87 @@ public PlannerContext getPlannerContext() {
       return _plannerContext;
     }
 
+    /// Builds the dispatchable plan. A plan that fails the `ARRAY_TO_MV` 
validation is planned again with the MV
+    /// aggregate (see [#replanWithMvAggregate]).
+    private DispatchableSubPlan toDispatchableSubPlanWithMvAggregateFallback() 
{
+      return withMvAggregateFallback(() -> toDispatchableSubPlan(_relRoot, 
_plannerContext));
+    }
+
+    /// Runs `planner`, which builds the dispatchable plan from [#_relRoot]. 
When it fails the `ARRAY_TO_MV` validation
+    /// and [#replanWithMvAggregate] applies, runs it again on the plan with 
the MV aggregate. If that also fails, the
+    /// error of the plan with the MV aggregate is thrown, with the first 
error as a suppressed exception.
+    private <T> T withMvAggregateFallback(Supplier<T> planner) {
+      try {
+        return planner.get();
+      } catch (QueryException e) {
+        replanWithMvAggregate(e);
+        try {
+          return planner.get();
+        } catch (QueryException e2) {
+          e2.addSuppressed(e);
+          throw e2;
+        }
+      }
+    }
+
+    /// The MV aggregate fallback: when the plan without the MV aggregate 
fails the `ARRAY_TO_MV` validation, plans the
+    /// query again with the MV aggregate on and replaces [#_relRoot]. 
Rethrows `e` when the fallback does not apply. If
+    /// the second plan fails, its error is thrown with `e` as a suppressed 
exception.
+    ///
+    /// The second plan starts from the SQL: it validates 
[#_mvAggregateFallbackQueryNode] with the validator of
+    /// [PlannerContext#enableMvAggregateFallback()], where `ARRAY_TO_MV` has 
a nullable type, then converts and
+    /// optimizes it. Only a query that fails without the MV aggregate 
changes. A query that passes the validation keeps
+    /// the plan and the types of a broker that predates the MV aggregate, so 
servers that predate it can run it.
+    private void replanWithMvAggregate(QueryException e) {
+      if (!(e instanceof 
ArrayToMvValidationVisitor.ArrayToMvValidationException) || 
!canFallBackToMvAggregate()) {
+        throw e;
+      }
+      SqlNode queryNode = _mvAggregateFallbackQueryNode;
+      // The validation changes the copy, so it can be used only once
+      _mvAggregateFallbackQueryNode = null;
+      _plannerContext.enableMvAggregateFallback();
+      try {
+        RelRoot unoptimizedRelRoot = toRelation(validate(queryNode, 
_plannerContext), _plannerContext);
+        _relRoot = 
unoptimizedRelRoot.withRel(optimizeAndConvert(unoptimizedRelRoot, 
_plannerContext));

Review Comment:
   I reproduced this end to end on `RowLevelSecurityIntegrationTest` at 
`15f9f05467`. User `user` has the RLS filter `AirlineID='19805'` on `mytable` 
and no filter on `mytable2`.
   
   ```sql
   SET enableNullHandling=true;
   SELECT ARRAY_TO_MV(s.DivAirports) AS v, COUNT(*) AS c
   FROM (SELECT DISTINCT AirlineID FROM mytable2) p
   LEFT JOIN mytable s ON p.AirlineID = s.AirlineID AND s.ArrDelay > 100
   GROUP BY ARRAY_TO_MV(s.DivAirports)
   HAVING ARRAY_TO_MV(s.DivAirports) IS NULL
   UNION ALL
   SELECT /*+ aggOptions(is_skip_leaf_stage_group_by='true') */ 
ARRAY_TO_MV(DivAirports), COUNT(*)
   FROM mytable2 WHERE ArrDelay > 1000 GROUP BY ARRAY_TO_MV(DivAirports)
   ```
   
   The null group counts the airlines that have no delayed flight in the 
visible rows of `mytable`.
   
   | Run | Null group | `rlsFiltersApplied` |
   |---|---|---|
   | `user` | none | `false` |
   | admin (no RLS) | none | `false` |
   | admin, with `AND s.AirlineID = 19805` in the `ON` (the correct result for 
`user`) | `(null, 13)` | `false` |
   | `user`, same query without the `HAVING` (control) | `(null, 13)` | `true` |
   
   So `user` gets the unfiltered result, and the response says that no RLS 
filter was applied.
   
   Two notes:
   - The query needs `enableNullHandling=true`. Without it, `IS NULL` is 
`PINOT_IS_NULL`, which Calcite does not fold. Then the first plan keeps 
`mytable`, and RLS works. A user can set the option in the query. The planner 
unit tests have null handling on (the test constructor of `QueryEnvironment` 
passes `true`), so they do not show this difference.
   - No nullable column is necessary. The `LEFT JOIN` makes the array nullable 
in the second plan.
   
   My review has a fix that also removes code: decide the fallback inside 
`compile()`. With it, this query returns `(null, 13)` for `user`, with 
`rlsFiltersApplied=true`.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to