JackieTien97 commented on code in PR #18982:
URL: https://github.com/apache/iotdb/pull/18982#discussion_r4345027620


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java:
##########
@@ -1573,9 +1576,11 @@ private void analyzeWhere(
     if (!queryStatement.hasWhere()) {
       return;
     }
+    Expression wherePredicate =
+        
PredicateUtils.predicateRemoveNot(queryStatement.getWhereCondition().getPredicate());

Review Comment:
   `predicateRemoveNot` throws when the `NOT` wraps something 
`ReversePredicateVisitor` can't invert. A BOOLEAN series, a function/UDF, `CASE 
WHEN` and a constant all fall through to 
`ReversePredicateVisitor#visitExpression`, which throws `new 
UnsupportedOperationException()` with no message.
   
   This now runs on every WHERE predicate (here, at L1608, and in 
`TemplatedAnalyze` at L269). On master, the queries below work because the 
conjunct is never pushed into a scan. With this PR they fail during analysis 
(setup in the review summary):
   
   ```sql
   -- non-aligned d1: the conjunct references two series, so it is never pushed 
down
   SELECT s1, b FROM root.rv.d1 WHERE NOT b OR s1 > 5
   --   master: t2, t4         this PR: 305 ... executeStatement failed. null
   
   SELECT s1, b FROM root.rv.d1 WHERE NOT (CASE WHEN s1 > 5 THEN true ELSE 
false END) OR s2 > 35
   --   master: t1, t3, t4     this PR: 305 ... null
   
   -- non-aligned template: predicates are never pushed down on this path
   SELECT s1, b FROM root.tn.* WHERE NOT b ALIGN BY DEVICE
   --   master: (2, root.tn.d1, 8, false), (1, root.tn.d2, 4, false)    this 
PR: 305 ... null
   ```
   
   `SELECT s1 FROM root.rv.d1 WHERE NOT b` already fails on master, but with a 
clear 301 message. With this PR it becomes a 305 with the message `null`, and 
the DataNode logs an ERROR stack:
   
   ```
   java.lang.UnsupportedOperationException
        at 
...ReversePredicateVisitor.visitExpression(ReversePredicateVisitor.java:45)
        at ...PredicateUtils.reversePredicate(PredicateUtils.java:289)
        at ...PredicateUtils.predicateRemoveNot(PredicateUtils.java:273)
        at ...AnalyzeVisitor.analyzeWhere(AnalyzeVisitor.java:1580)
   ```
   
   This also means these predicates never reach the new 
`PredicateCanPushDownToSourceChecker#visitLogicNotExpression` branch. So the 
description's claim that non-invertible `NOT` predicates remain in 
`FilterOperator` doesn't hold yet.
   
   Suggestion: when the operand can't be inverted, keep `NOT x` instead of 
throwing (for example, return `new LogicNotExpression(x)`), and let the new 
checker keep it in `FilterOperator`.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/TemplatedAggregationAnalyze.java:
##########
@@ -293,7 +293,8 @@ private static boolean analyzeHaving(Analysis analysis, 
QueryStatement queryStat
 
     Set<String> measurementSet = new HashSet<>(analysis.getMeasurementList());
     Set<Expression> aggregationExpressions = 
analysis.getAggregationExpressions();
-    Expression havingExpression = 
queryStatement.getHavingCondition().getPredicate();
+    Expression havingExpression =
+        
PredicateUtils.predicateRemoveNot(queryStatement.getHavingCondition().getPredicate());

Review Comment:
   Same situation as the ALIGN BY DEVICE HAVING path. The templated HAVING 
expression is never bound by a root-splitting binder and is never pushed down, 
so composite `NOT` already works on master. For example, `SELECT count(s1) FROM 
root.ta.* GROUP BY ([1,3),1ms) HAVING NOT (count(s1) > 0 AND last_value(s1) > 
5) ALIGN BY DEVICE` returns the same rows on both.
   
   With this change, the following aligned-template query fails:
   
   ```sql
   SELECT count(s1) FROM root.ta.* GROUP BY ([1,3),1ms) HAVING NOT 
last_value(b) ALIGN BY DEVICE
   --   master: (2, root.ta.d1, 1), (1, root.ta.d2, 1)    this PR: 305 ... null
   ```
   
   I'd suggest reverting this hunk as well.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java:
##########
@@ -1573,9 +1576,11 @@ private void analyzeWhere(
     if (!queryStatement.hasWhere()) {
       return;
     }
+    Expression wherePredicate =
+        
PredicateUtils.predicateRemoveNot(queryStatement.getWhereCondition().getPredicate());
     List<Expression> conJunctions =
         ExpressionAnalyzer.bindSchemaForPredicate(
-            queryStatement.getWhereCondition().getPredicate(),
+            wherePredicate,

Review Comment:
   `predicateRemoveNot` doesn't remove every `NOT` when negations are nested. 
`ReversePredicateVisitor#visitLogicNotExpression` returns the child unchanged, 
so `predicateRemoveNot(NOT (NOT (NOT (A AND B))))` is still `NOT (A AND B)`.
   
   That residual `NOT` reaches `bindSchemaForPredicate` at the root. There, 
`isRoot` leaks through the `NOT` because 
`CartesianProductVisitor#getResultsFromChild` reuses the same context. The 
`AND` is therefore split, and the predicate becomes `NOT A AND NOT B`. On 
master this query errors out; with this PR it returns a wrong result and no 
error:
   
   ```sql
   SELECT s1, s2 FROM root.rv.d1 WHERE NOT (NOT (NOT (s1 > 5 AND s2 > 25)))
   --   expected (same as WHERE s1 <= 5 OR s2 <= 25): t1, t2, t3
   --   this PR: t1 only
   ```
   
   The HAVING variant, `... HAVING NOT (NOT (NOT (count(s1) > 1 AND sum(s2) > 
50)))`, returns an empty set instead of one row.
   
   Suggestion: have `visitLogicNotExpression` return 
`PredicateUtils.predicateRemoveNot(logicNotExpression.getExpression())`.
   
   Fixing how `isRoot` propagates through unary expressions in 
`BindSchemaForPredicateVisitor` would go further and remove this whole class of 
bugs at the source. It is also the root cause of the composite `NOT` issue this 
PR works around.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java:
##########
@@ -1030,10 +1030,12 @@ private void analyzeHavingBase(
       ISchemaTree schemaTree,
       UnaryOperator<Expression> havingExpressionAnalyzer,
       MPPQueryContext queryContext) {
+    Expression havingPredicate =
+        
PredicateUtils.predicateRemoveNot(queryStatement.getHavingCondition().getPredicate());

Review Comment:
   HAVING hits the same exception. HAVING is never pushed down, so `HAVING NOT 
<boolean aggregation>` works on master:
   
   ```sql
   SELECT count(s1) FROM root.rv.d1 GROUP BY ([1,5),1ms) HAVING NOT 
last_value(b)
   --   master: (2, 1), (4, 1)    this PR: 305 ... null
   ```
   
   That said, the rewrite here does fix a real bug. 
`bindSchemaForPredicate(..., isRoot = true)` splits `NOT (A AND B)` into `NOT A 
AND NOT B`, because `isRoot` passes through the `NOT`. As a result, these two 
queries return an empty set on master and the correct rows with this PR:
   
   ```sql
   SELECT count(s1), sum(s2) FROM root.rv.d1 GROUP BY ([1,5),2ms) HAVING NOT 
(count(s1) > 1 AND sum(s2) > 50)
   --   master: empty    this PR: (1, 2, 30.0)
   SELECT count(s1) FROM root.rv.* GROUP BY ([1,5),2ms), LEVEL=1 HAVING NOT 
(count(s1) > 4 AND count(s1) < 6)
   --   master: empty    this PR: (1, 6), (3, 2)
   ```
   
   So this hunk should stay, just with a `NOT` removal that doesn't throw.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java:
##########
@@ -1110,7 +1112,8 @@ private void analyzeHaving(
     Map<IDeviceID, Set<Expression>> deviceToOutputExpressions =
         analysis.getDeviceToOutputExpressions();
 
-    Expression havingExpression = 
queryStatement.getHavingCondition().getPredicate();
+    Expression havingExpression =
+        
PredicateUtils.predicateRemoveNot(queryStatement.getHavingCondition().getPredicate());

Review Comment:
   I don't think this hunk is needed. 
`ConcatDeviceAndBindSchemaForHavingVisitor` doesn't split a root `AND`, and 
HAVING is never pushed down, so composite `NOT` already gives the right answer 
on master. For example, `SELECT count(s1), sum(s2) FROM root.rv.d1 GROUP BY 
([1,5),2ms) HAVING NOT (count(s1) > 1 AND sum(s2) > 50) ALIGN BY DEVICE` 
returns `(1, root.rv.d1, 2, 30.0)` on both master and this PR.
   
   The only observable effect of the rewrite here is the regression:
   
   ```sql
   SELECT count(s1) FROM root.rv.d1 GROUP BY ([1,5),1ms) HAVING NOT 
last_value(b) ALIGN BY DEVICE
   --   master: (2, root.rv.d1, 1), (4, root.rv.d1, 1)    this PR: 305 ... null
   ```
   
   I'd suggest reverting this hunk.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/expression/visitor/logical/PredicateCanPushDownToSourceChecker.java:
##########
@@ -37,4 +38,11 @@ public Boolean visitIsNullExpression(IsNullExpression 
isNullExpression, Void con
   public Boolean visitLeafOperand(LeafOperand leafOperand, Void context) {
     return Boolean.TRUE;
   }
+
+  @Override
+  public Boolean visitLogicNotExpression(LogicNotExpression 
logicNotExpression, Void context) {
+    // TsFile filters cannot represent NOT. Predicates that cannot be inverted 
during analysis must

Review Comment:
   Nit, not blocking:
   
   1. Because of the `predicateRemoveNot` issue in `AnalyzeVisitor`, analysis 
throws before predicates that can't be inverted ever reach this checker. So 
this comment doesn't describe the current behaviour.
   2. `LogicalAndVisitor` recurses into comparison operands, function arguments 
and `CASE` branches. As a result, this also stops push-down for conjuncts with 
a *nested* `NOT`, e.g. `(NOT (s1 > 5)) = true`.
   
      Those were safe before. `PredicatePushIntoScanChecker` only reaches 
`visitLogicNotExpression` through the top-level AND/OR structure, so for these 
conjuncts it returned `false` rather than throwing. `OperatorTreeGenerator` 
then evaluated them in a filter operator directly above the scan.
   
      This costs push-down, not correctness. Still, returning `FALSE` only for 
`NOT`s in the logical (AND/OR) structure would be more precise.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java:
##########
@@ -1599,13 +1604,11 @@ private Expression analyzeWhereSplitByDevice(
       final PartialPath devicePath,
       final ISchemaTree schemaTree,
       final MPPQueryContext queryContext) {
+    Expression wherePredicate =
+        
PredicateUtils.predicateRemoveNot(queryStatement.getWhereCondition().getPredicate());

Review Comment:
   `analyzeDeviceToWhere` calls `analyzeWhereSplitByDevice` once per device, so 
the same predicate is rewritten N times for N devices. `predicateRemoveNot` 
also rebuilds the whole AND/OR spine even when the predicate contains no `NOT`. 
At a minimum, this should be hoisted out of the device loop.
   
   More generally, I'd suggest normalizing `NOT` once, at the start of 
`analyzeGlobalTimeFilter`, instead of at six consumer sites; 
`analyzeGlobalTimeFilter` already installs a working predicate through 
`mutationJournal`. Besides removing the duplication, this would let 
`extractGlobalTimePredicate` see the rewritten predicate.
   
   Today, `WHERE NOT (time > 2 OR s1 > 5)` is equivalent to `time <= 2 AND s1 
<= 5`. But extraction runs on the original predicate, so the `LOGIC_NOT` branch 
yields no global time predicate. As a result, partition fetching 
(`getTimePartitionSlotList(context.getGlobalTimeFilter(), ...)`) covers the 
full time range.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/AnalyzeVisitor.java:
##########
@@ -1573,9 +1576,11 @@ private void analyzeWhere(
     if (!queryStatement.hasWhere()) {
       return;
     }
+    Expression wherePredicate =

Review Comment:
   For FLOAT/DOUBLE NaN, inverting the comparison is not equivalent to `NOT`. 
Both the column transformers and the pushed-down TsFile filters evaluate every 
ordered comparison on NaN to false. So `NOT (NaN > 5)` is true, but `NaN <= 5` 
is false; the same holds for `<`, `<=` and `>=`.
   
   The rewrite runs before schema binding, so it can't take the data type into 
account. On master, the queries below are evaluated in `FilterOperator`; with 
this PR they silently return different rows:
   
   ```sql
   CREATE TIMESERIES root.rv.d3.x WITH DATATYPE=DOUBLE, ENCODING=PLAIN;
   CREATE TIMESERIES root.rv.d3.y WITH DATATYPE=INT32, ENCODING=PLAIN;
   INSERT INTO root.rv.d3(time, x, y) VALUES (1, NaN, 1);
   INSERT INTO root.rv.d3(time, x, y) VALUES (2, 3.0, 2);
   INSERT INTO root.rv.d3(time, x, y) VALUES (3, 10.0, 3);
   
   SELECT x, y FROM root.rv.d3 WHERE NOT (x > 5 OR y > 2)
   --   master: t1 (NaN, 1), t2 (3.0, 2)    this PR: t2 only
   SELECT x, y FROM root.rv.d3 WHERE NOT (x + y > 5)
   --   master: t1, t2                      this PR: t2 only
   ```
   
   I see two options:
   
   - Don't invert ordered comparisons whose operands may be FLOAT/DOUBLE. This 
needs type information, so the rewrite would have to happen after binding.
   - Treat this as an intentional semantic change: document it and add tests.
   
   NaN filtering is already covered by tests such as 
`IoTDBFilterIT#testFilterNaN`.



##########
integration-test/src/test/java/org/apache/iotdb/db/it/query/IoTDBPredicateConversionTreeIT.java:
##########
@@ -57,7 +61,15 @@ public class IoTDBPredicateConversionTreeIT {
         "INSERT INTO " + DEVICE_ID + "(time, int32_col, int64_col) VALUES(7, 
null, null)",
         "INSERT INTO " + DEVICE_ID + "(time, int32_col, int64_col) VALUES(8, 
1000, 1000)",
         "INSERT INTO " + DEVICE_ID + "(time, int32_col, int64_col) VALUES(9, 
-29, -29)",
-        "INSERT INTO " + DEVICE_ID + "(time, int32_col, int64_col) VALUES(10, 
-30, -30)"
+        "INSERT INTO " + DEVICE_ID + "(time, int32_col, int64_col) VALUES(10, 
-30, -30)",
+        "CREATE DATABASE " + TEMPLATE_DATABASE_NAME,
+        "CREATE DEVICE TEMPLATE not_predicate_template (s1 INT64)",

Review Comment:
   This template is non-aligned. For non-aligned templates, `PredicatePushDown` 
never pushes the predicate down: `visitSeriesScan` and `visitFullOuterTimeJoin` 
return early when `isBuildPlanUseTemplate()`. So `testTemplatedWhereWithNot` 
already passes on master and doesn't exercise the `TemplatedAnalyze` change. 
The failure only reproduces with an aligned template:
   
   ```sql
   CREATE DEVICE TEMPLATE ta_t ALIGNED (s1 INT64, b BOOLEAN);
   SET DEVICE TEMPLATE ta_t TO root.ta.d1;
   SET DEVICE TEMPLATE ta_t TO root.ta.d2;
   INSERT INTO root.ta.d1(time, s1, b) ALIGNED VALUES (1, 3, true);
   INSERT INTO root.ta.d1(time, s1, b) ALIGNED VALUES (2, 8, false);
   INSERT INTO root.ta.d2(time, s1, b) ALIGNED VALUES (1, 4, false);
   INSERT INTO root.ta.d2(time, s1, b) ALIGNED VALUES (2, 9, true);
   
   SELECT s1 FROM root.ta.* WHERE NOT (s1 > 5) ALIGN BY DEVICE
   --   master: 301 This predicate contains a not!         this PR: (1, 
root.ta.d1, 3), (1, root.ta.d2, 4)
   SELECT count(s1) FROM root.ta.* WHERE NOT (s1 > 5) ALIGN BY DEVICE
   --   master: 305 ... This predicate contains a not!     this PR: 
(root.ta.d1, 1), (root.ta.d2, 1)
   ```
   
   Could you use an `ALIGNED` template here, or add a second case that uses one?



##########
integration-test/src/test/java/org/apache/iotdb/db/it/query/IoTDBPredicateConversionTreeIT.java:
##########
@@ -290,4 +302,54 @@ public void testDoubleLiteralOverflow() {
         expectedHeaderInt64,
         allInt64Rows);
   }
+
+  @Test
+  public void testWhereAndHavingWithNot() {
+    String[] expectedWhereHeader = new String[] {TIMESTAMP_STR, DEVICE_ID + 
".int32_col"};
+    resultSetEqualTest(
+        "SELECT int32_col FROM " + DEVICE_ID + " WHERE NOT (int32_col > 20)",
+        expectedWhereHeader,
+        new String[] {"1,20,", "9,-29,", "10,-30,"});
+    resultSetEqualTest(
+        "SELECT int32_col FROM " + DEVICE_ID + " WHERE NOT (int32_col > 20 AND 
int32_col < 40)",
+        expectedWhereHeader,
+        new String[] {"1,20,", "8,1000,", "9,-29,", "10,-30,"});
+
+    String[] expectedHavingHeader = new String[] {TIMESTAMP_STR, 
count(DEVICE_ID + ".int32_col")};
+    resultSetEqualTest(
+        "SELECT count(int32_col) FROM "
+            + DEVICE_ID
+            + " GROUP BY ([1,11),2ms) HAVING NOT (count(int32_col) > 1)",

Review Comment:
   Both `HAVING NOT (count(int32_col) > 1)` cases (with and without ALIGN BY 
DEVICE) already pass on master, so they don't cover the HAVING change. The 
HAVING bug this PR actually fixes is a composite `NOT` on the non-ALIGN BY 
DEVICE and GROUP BY LEVEL paths (see the comment on `analyzeHavingBase`).
   
   It would be good to add:
   - composite HAVING, e.g. `HAVING NOT (count(...) > 1 AND sum(...) > 50)`, 
with and without `GROUP BY LEVEL`
   - non-invertible `NOT`: `WHERE NOT bool_series`, `WHERE NOT bool_series OR 
s1 > 5`, `HAVING NOT last_value(bool_series)`
   - nested negation: `WHERE NOT (NOT (NOT (A AND B)))`
   - a FLOAT/DOUBLE series containing NaN under `NOT`
   
   Any of the last three would have caught the issues raised in the other 
comments. `PredicateUtilsTest#testLogicNotCannotPushDownToSource` covers only 
the checker, not the analyzer path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to