JackieTien97 commented on code in PR #19250:
URL: https://github.com/apache/iotdb/pull/19250#discussion_r4558847294


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/TableDistributedPlanGenerator.java:
##########
@@ -2296,6 +2314,121 @@ static boolean isInputOrderedByTimeAscending(
     return true;
   }
 
+  private boolean isChildTimeOrderedPerGroup(AggregationNode node, PlanNode 
child) {
+    DeviceTableScanNode scan = findOrderedDeviceScan(child);
+    if (scan == null) {
+      return false;
+    }
+    return isDeviceScanTimeOrderedPerGroup(
+        node.getStep(),
+        node.getGroupingKeys(),
+        child,
+        scan,
+        analysis.getTableColumnSchema(scan.getQualifiedObjectName()));
+  }
+
+  static boolean isDeviceScanTimeOrderedPerGroup(
+      AggregationNode.Step step,
+      List<Symbol> groupingKeys,
+      PlanNode child,
+      DeviceTableScanNode scan,
+      Map<Symbol, ColumnSchema> tableColumnSchema) {
+    if (step != AggregationNode.Step.SINGLE || scan.getScanOrder() != 
Ordering.ASC) {
+      return false;
+    }
+    int deviceEntryCount = scan.getDeviceEntryCount();
+    if (deviceEntryCount <= 0) {
+      return false;
+    }
+    if (deviceEntryCount == 1) {
+      return true;
+    }
+    if (tableColumnSchema == null) {
+      return false;
+    }
+    Set<String> tagColumnNames = new HashSet<>();
+    for (ColumnSchema columnSchema : tableColumnSchema.values()) {
+      if (columnSchema.getColumnCategory() == TsTableColumnCategory.TAG) {
+        tagColumnNames.add(columnSchema.getName());
+      }
+    }
+    Map<Symbol, ColumnSchema> scanColumnSchema = scan.getAssignments();
+    Set<String> groupedColumnNames = new HashSet<>();
+    for (Symbol symbol : resolveSymbolsToScanLevel(child, groupingKeys)) {
+      ColumnSchema columnSchema = scanColumnSchema.get(symbol);
+      if (columnSchema != null) {
+        groupedColumnNames.add(columnSchema.getName());
+      }
+    }
+    return groupedColumnNames.containsAll(tagColumnNames);
+  }
+
+  private static boolean isTimeArgumentNativeTime(
+      AggregationNode.Aggregation aggregation, PlanNode child, 
DeviceTableScanNode scan) {
+    if (scan == null || aggregation.getArguments().size() < 2) {
+      return false;
+    }
+    Expression timeArgument = aggregation.getArguments().get(1);

Review Comment:
   The time argument is hard-coded as `getArguments().get(1)` here and in 
`isInputOrderedByTimeAscending`, but the comment on `isOrderedInputFunction` 
asks future families such as integral to be added to that list. In #19227, 
`integral(value, time_unit, time)` takes the time column as its third argument, 
and that PR rewrites the same block of `isInputOrderedByTimeAscending` to pick 
the index per function, so the two PRs will conflict there.
   
   If INTEGRAL is only added to `isOrderedInputFunction`, as the comment 
suggests, both checks will look at the time-unit argument instead of the time 
column. The ordered integral accumulators would then never be selected. Results 
would still be correct, but no test would notice. Could we extract one helper 
that returns the time argument for each function, use it in both checks, and 
mention it in that comment?



##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/RateOrderedPlannerTest.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner.distribute;
+
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.planner.node.AggregationNode;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.TestMetadata;
+import org.apache.iotdb.db.queryengine.plan.relational.planner.PlanTester;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+public class RateOrderedPlannerTest {
+
+  @Test
+  public void testSingleDeviceQueryEnablesOrderedInput() {
+    AggregationNode aggregation =
+        planAggregation(
+            "SELECT irate(s1, time) AS ir FROM table1 "
+                + "WHERE tag1='shanghai' AND tag2='A3' AND tag3='YY'");
+    assertNotNull(aggregation);
+    assertTrue(hasOrderedInput(aggregation));
+  }
+
+  @Test
+  public void testMultiDeviceQueryKeepsNaiveInput() {
+    AggregationNode aggregation = planAggregation("SELECT irate(s1, time) AS 
ir FROM table1");

Review Comment:
   This case can't fail. `table1` spans three DataRegions in `TestMetadata` 
(`DATA_REGION_GROUP_1` to `DATA_REGION_GROUP_3`), so the aggregation is split 
into PARTIAL/FINAL and `planAggregation` returns the FINAL (or a PARTIAL) node, 
neither of which is ever marked ordered.
   
   So the new wiring in `visitAggregation` → `isChildTimeOrderedPerGroup` has 
no effective test; `RateOrderedInputTest` calls the static helpers directly. I 
checked this with a mutant that makes `isChildTimeOrderedPerGroup` return 
`true` for every SINGLE aggregation, skipping the device-count, TAG-coverage, 
and scan-order checks. All 10 new tests still pass, while the planner then 
marks these as ordered: `... WHERE tag1='shanghai' GROUP BY tag1`, a global 
aggregation with `WHERE tag1='shanghai'`, and an `ORDER BY time DESC LIMIT 10` 
subquery. On typical data those queries would then fail at runtime with an 
ordered-input violation or a duplicate-timestamp error.
   
   Suggestion: DEVICE_3 and DEVICE_4 are both in `DATA_REGION_GROUP_2`, so 
`WHERE tag1='shanghai'` gives a SINGLE aggregation over a two-device scan. 
Assert that `GROUP BY tag1, tag2, tag3` is ordered and that `GROUP BY tag1`, 
the global aggregation, and the DESC subquery are not. Also assert `step == 
SINGLE` in `planAggregation`, so a case can't silently pick a FINAL or PARTIAL 
node.



##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/RateOrderedInputTest.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner.distribute;
+
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeId;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.metadata.ColumnMetadata;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.metadata.ColumnSchema;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.metadata.QualifiedObjectName;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.planner.Assignments;
+import org.apache.iotdb.commons.queryengine.plan.relational.planner.Symbol;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.planner.node.AggregationNode;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.planner.node.ProjectNode;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.sql.ast.SymbolReference;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory;
+import org.apache.iotdb.db.queryengine.plan.relational.metadata.DeviceEntry;
+import 
org.apache.iotdb.db.queryengine.plan.relational.planner.node.DeviceTableScanNode;
+import org.apache.iotdb.db.queryengine.plan.statement.component.Ordering;
+
+import org.apache.tsfile.read.common.type.StringType;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+import static 
org.apache.iotdb.commons.queryengine.plan.relational.planner.node.AggregationNode.Step.PARTIAL;
+import static 
org.apache.iotdb.commons.queryengine.plan.relational.planner.node.AggregationNode.Step.SINGLE;
+import static 
org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory.FIELD;
+import static 
org.apache.iotdb.commons.schema.table.column.TsTableColumnCategory.TAG;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
+public class RateOrderedInputTest {
+
+  private static final Symbol TAG1 = new Symbol("tag1");
+  private static final Symbol TAG2 = new Symbol("tag2");
+  private static final Symbol S1 = new Symbol("s1");
+
+  @Test
+  public void testSingleDeviceIsOrderedPerGroup() {
+    assertTrue(isOrderedPerGroup(SINGLE, Collections.emptyList(), scan(1)));
+  }
+
+  @Test
+  public void testGroupedByAllTagsIsOrderedPerGroup() {
+    assertTrue(isOrderedPerGroup(SINGLE, Arrays.asList(TAG1, TAG2), scan(2)));
+  }
+
+  @Test
+  public void testGroupedBySubsetOfTagsIsNotOrderedPerGroup() {
+    assertFalse(isOrderedPerGroup(SINGLE, Collections.singletonList(TAG1), 
scan(2)));
+  }
+
+  @Test
+  public void testDescendingScanIsNotOrderedPerGroup() {
+    DeviceTableScanNode scan = scan(1);
+    scan.setScanOrder(Ordering.DESC);
+    assertFalse(isOrderedPerGroup(SINGLE, Collections.emptyList(), scan));
+  }
+
+  @Test
+  public void testNonSingleStepIsNotOrderedPerGroup() {
+    assertFalse(isOrderedPerGroup(PARTIAL, Collections.emptyList(), scan(1)));
+  }
+
+  @Test
+  public void testGroupingKeysAreResolvedThroughProjections() {
+    DeviceTableScanNode scan = scan(2);
+    Symbol tag1Out = new Symbol("tag1_out");
+    Symbol tag2Out = new Symbol("tag2_out");
+    ProjectNode project =
+        new ProjectNode(
+            new PlanNodeId("project"),
+            scan,
+            Assignments.builder()
+                .put(tag1Out, new SymbolReference("tag1"))
+                .put(tag2Out, new SymbolReference("tag2"))
+                .build());
+    assertTrue(isOrderedPerGroup(SINGLE, Arrays.asList(tag1Out, tag2Out), 
project));
+  }
+
+  @Test
+  public void testOrderChangingChildIsRejected() {
+    
assertNull(TableDistributedPlanGenerator.findOrderedDeviceScan(Mockito.mock(PlanNode.class)));

Review Comment:
   This passes a bare mock with no scan underneath, so it would still pass if 
`findOrderedDeviceScan` were changed to walk through any single-child node, 
such as a `LimitNode` or a `SortNode(time DESC)`; `Aggregation → Sort(time 
DESC) → Scan` would then be treated as ordered. Could this wrap a real 
`DeviceTableScanNode` in a `SortNode`, a `LimitNode`, and a single-child 
`CollectNode` and assert `null` for each? It would also help to cover the 
excluded `TreeDeviceViewScanNode`, `AggregationTableScanNode`, and 
`ExternalTsFileScanNode` cases.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/TableDistributedPlanGenerator.java:
##########
@@ -2246,6 +2251,15 @@ private boolean containsUnorderedCollect(PlanNode node) {
 
   private static AggregationNode withRateFunctionInputOrdering(
       AggregationNode node, OrderingScheme childOrdering) {
+    return withRateFunctionInputOrdering(node, childOrdering, false, null);
+  }
+
+  private static AggregationNode withRateFunctionInputOrdering(
+      AggregationNode node,
+      OrderingScheme childOrdering,
+      boolean childTimeOrderedPerGroup,
+      PlanNode child) {
+    DeviceTableScanNode scan = childTimeOrderedPerGroup ? 
findOrderedDeviceScan(child) : null;

Review Comment:
   Minor: `findOrderedDeviceScan(child)` is computed in 
`isChildTimeOrderedPerGroup` and again here. `isChildTimeOrderedPerGroup` also 
runs for every single-child aggregation, including plain `count`/`sum`, and 
iterates the whole table schema to collect TAG names each time. Passing a 
nullable `DeviceTableScanNode` instead of the `boolean` + `child` pair, 
computed once and only when some aggregation is an ordered-input function, 
would simplify this. It would also make the `scan == null` check in 
`isTimeArgumentNativeTime` unnecessary.



##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/RateOrderedPlannerTest.java:
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.plan.relational.planner.distribute;
+
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNode;
+import 
org.apache.iotdb.commons.queryengine.plan.relational.planner.node.AggregationNode;
+import org.apache.iotdb.db.queryengine.plan.relational.analyzer.TestMetadata;
+import org.apache.iotdb.db.queryengine.plan.relational.planner.PlanTester;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+public class RateOrderedPlannerTest {
+
+  @Test
+  public void testSingleDeviceQueryEnablesOrderedInput() {
+    AggregationNode aggregation =
+        planAggregation(
+            "SELECT irate(s1, time) AS ir FROM table1 "

Review Comment:
   All planner cases use the two-argument `irate`. Could we also cover:
   - the four-argument `rate`, `increase`, and `delta` on the new path;
   - a time argument that reaches the scan through an alias, e.g. `SELECT 
irate(v, t) FROM (SELECT s1 AS v, time AS t FROM table1 WHERE ...)`. 
`isTimeArgumentNativeTime` resolves it, but nothing tests it; the projection 
test in `RateOrderedInputTest` only covers grouping keys;
   - an IT with several devices in one region grouped by all TAG columns. Today 
only the first query of `IoTDBTableAggregationIT.rateFunctionsNormalTest` (a 
single device) reaches the new path at runtime.
   
   For reference, I ran these cases on a local 1C1D with one DataRegion and 
four devices (sequence and unsequence files, memtable, two time partitions). 
Results with `time` matched those with an equal INT64 column (naive path) 
exactly, so they would make good regression queries.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/planner/distribute/TableDistributedPlanGenerator.java:
##########
@@ -2296,6 +2314,121 @@ static boolean isInputOrderedByTimeAscending(
     return true;
   }
 
+  private boolean isChildTimeOrderedPerGroup(AggregationNode node, PlanNode 
child) {
+    DeviceTableScanNode scan = findOrderedDeviceScan(child);
+    if (scan == null) {
+      return false;
+    }
+    return isDeviceScanTimeOrderedPerGroup(
+        node.getStep(),
+        node.getGroupingKeys(),
+        child,
+        scan,
+        analysis.getTableColumnSchema(scan.getQualifiedObjectName()));
+  }
+
+  static boolean isDeviceScanTimeOrderedPerGroup(
+      AggregationNode.Step step,
+      List<Symbol> groupingKeys,
+      PlanNode child,
+      DeviceTableScanNode scan,
+      Map<Symbol, ColumnSchema> tableColumnSchema) {
+    if (step != AggregationNode.Step.SINGLE || scan.getScanOrder() != 
Ordering.ASC) {
+      return false;
+    }
+    int deviceEntryCount = scan.getDeviceEntryCount();
+    if (deviceEntryCount <= 0) {
+      return false;
+    }
+    if (deviceEntryCount == 1) {
+      return true;
+    }
+    if (tableColumnSchema == null) {
+      return false;
+    }
+    Set<String> tagColumnNames = new HashSet<>();

Review Comment:
   Nit: this is the third implementation of "the keys cover every TAG column". 
`ParallelizeGrouping.visitDeviceTableScan` compares symbols and 
`TransformSortToStreamSort.isOrderByAllIdsAndTime` counts TAG columns, while 
this compares column names. A shared helper would keep the three consistent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to