Copilot commented on code in PR #19035:
URL: https://github.com/apache/iotdb/pull/19035#discussion_r4462672866


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -423,13 +427,51 @@ public ExecutionResult executeForTreeModel(
     }
   }
 
+  /**
+   * The storage layer assumes the rows of a tablet are sorted by time: the 
sequence/unsequence
+   * classification, the time boundaries of the tsfile resource, the ttl check 
and the time
+   * partition routing all rely on it. The child operator of an {@code INSERT 
INTO ... SELECT} does
+   * not guarantee that order, and REST/MQTT build the statement from the 
request as received, so
+   * restore it here: this runs on the coordinator, before the tablet is 
routed and before the write
+   * lock is taken. A tablet that is already sorted only costs one pass over 
its timestamps.
+   */
+  private static void sortTabletStatementIfNecessary(final Statement 
statement) {
+    // The pipe receiver and the write-back sink wrap the statement before 
submitting it.
+    final Statement target =
+        statement instanceof PipeEnrichedStatement
+            ? ((PipeEnrichedStatement) statement).getInnerStatement()
+            : statement;
+    if (target instanceof InsertTabletStatement) {
+      sortTabletStatement((InsertTabletStatement) target);
+    } else if (target instanceof InsertMultiTabletsStatement) {
+      for (final InsertTabletStatement tablet :
+          ((InsertMultiTabletsStatement) 
target).getInsertTabletStatementList()) {
+        sortTabletStatement(tablet);
+      }
+    }
+  }
+
+  private static void sortTabletStatement(final InsertTabletStatement tablet) {
+    if (tablet.getColumnCategories() == null) {
+      // A tree-model tablet holds a single device: sorting by time is all 
that is needed.
+      new InsertEventSorter(tablet).sortByTimestampIfNecessary();
+    } else {
+      // A table-model tablet can hold several devices. Keeping them together 
preserves the device
+      // segments the storage layer splits the tablet into, so only the times 
inside a device are
+      // reordered.
+      new 
TableModelTabletEventSorter(tablet).sortByDeviceIdAndTimestampIfNecessary();

Review Comment:
   Grouping a session tablet by device changes the row order used for RPC 
redirection. `RelationalInsertTabletNode.splitByReplicaSet()` builds 
`redirectNodeList` in this reordered order, but 
`Session.insertRelationalTabletOnce()` and `insertRelationalTabletByGroup()` 
associate endpoint i with device i in the original client tablet. For an 
already time-sorted request containing devices [b, a] on different leaders, the 
response now contains [leader(a), leader(b)], poisoning both client 
leader-cache entries and misrouting subsequent writes. Preserve the permutation 
and restore row-indexed response metadata to request order before returning it.



##########
integration-test/src/test/java/org/apache/iotdb/relational/it/insertquery/IoTDBInsertQueryIT.java:
##########
@@ -167,6 +167,83 @@ public void testSimple() throws SQLException {
     }
   }
 
+  /**
+   * End-to-end check for an {@code INSERT INTO ... SELECT} whose source 
tablet is not in ascending
+   * time order. The destination device is flushed at time 39, then the query 
feeds it another
+   * tablet: on the unfixed code the row at time 15 is classified as sequence 
data, a sequence
+   * TsFile overlapping [10,39] is created, and the query returns 15 after 39 
(or twice).
+   *
+   * <p>The out-of-order input depends on the nested-loop cross join emitting 
the rows in probe
+   * order, i.e. {@code [100, 15]}. If the optimizer ever emits those rows in 
time order this test
+   * no longer reproduces the situation and would pass trivially; the guards 
that do not depend on
+   * the plan shape are {@code InsertTabletStatementGeneratorTest} (the tablet 
is sorted before it
+   * leaves the engine) and {@code
+   * DataRegionTest#testOutOfOrderTabletKeepsSequenceSpaceNonOverlapped} (the 
storage-layer
+   * classification of an unsorted tablet).
+   *
+   * <p>The first sequence file also has to hold several pages (see {@code
+   * max_number_of_points_in_page} in {@link #setUpClass()}): with a single 
page the overlapping
+   * page is merged before the earlier pages are emitted, which hides the 
wrong order as well.

Review Comment:
   The 30 baseline rows do not establish the multi-page layout needed to expose 
this regression: `setUpClass()` never lowers the page-point limit mentioned 
here. With a single page, the unfixed reader can merge the overlap before 
emitting baseline data, so the assertions can pass without the sorting fix. Set 
`setMaxNumberOfPointsInPage(10)` before cluster startup and verify that the 
regression fails without sorting.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -423,13 +427,51 @@ public ExecutionResult executeForTreeModel(
     }
   }
 
+  /**
+   * The storage layer assumes the rows of a tablet are sorted by time: the 
sequence/unsequence
+   * classification, the time boundaries of the tsfile resource, the ttl check 
and the time
+   * partition routing all rely on it. The child operator of an {@code INSERT 
INTO ... SELECT} does
+   * not guarantee that order, and REST/MQTT build the statement from the 
request as received, so
+   * restore it here: this runs on the coordinator, before the tablet is 
routed and before the write
+   * lock is taken. A tablet that is already sorted only costs one pass over 
its timestamps.
+   */
+  private static void sortTabletStatementIfNecessary(final Statement 
statement) {
+    // The pipe receiver and the write-back sink wrap the statement before 
submitting it.
+    final Statement target =
+        statement instanceof PipeEnrichedStatement
+            ? ((PipeEnrichedStatement) statement).getInnerStatement()
+            : statement;
+    if (target instanceof InsertTabletStatement) {
+      sortTabletStatement((InsertTabletStatement) target);
+    } else if (target instanceof InsertMultiTabletsStatement) {
+      for (final InsertTabletStatement tablet :
+          ((InsertMultiTabletsStatement) 
target).getInsertTabletStatementList()) {
+        sortTabletStatement(tablet);
+      }
+    }
+  }
+
+  private static void sortTabletStatement(final InsertTabletStatement tablet) {
+    if (tablet.getColumnCategories() == null) {
+      // A tree-model tablet holds a single device: sorting by time is all 
that is needed.
+      new InsertEventSorter(tablet).sortByTimestampIfNecessary();
+    } else {
+      // A table-model tablet can hold several devices. Keeping them together 
preserves the device
+      // segments the storage layer splits the tablet into, so only the times 
inside a device are
+      // reordered.
+      new 
TableModelTabletEventSorter(tablet).sortByDeviceIdAndTimestampIfNecessary();

Review Comment:
   When a pipe-backed table statement needs reordering, its 
`InsertTabletStatement(InsertTabletNode)` constructor shares the source node's 
outer column and bitmap arrays. The sorter replaces timestamps only on the 
statement but changes those shared containers, leaving the source node with 
mismatched timestamp/value pairs that another pipe can replicate. Give the 
statement private column and bitmap containers before applying the permutation.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -676,6 +718,7 @@ private IQueryExecution createQueryExecutionForTableModel(
       Metadata metadata,
       long timeOut,
       long startTime) {
+    sortTabletStatementIfNecessary(statement);

Review Comment:
   Sorting before device-schema validation changes which attribute update wins: 
for one device with input rows `(100, attr='old'), (15, attr='new')`, 
`InsertTablet.getAttributeValueList()` now selects 'old' instead of the last 
non-null input value, 'new'. Attributes are device metadata, not timestamped 
fields. Preserve their original-order aggregation before sorting, for example 
by sorting after device-schema validation removes attribute columns but before 
partition routing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to