ColinLeeo commented on code in PR #19035:
URL: https://github.com/apache/iotdb/pull/19035#discussion_r4463635832


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/InsertTabletStatementGenerator.java:
##########
@@ -112,6 +114,8 @@ public InsertTabletStatement 
constructInsertTabletStatement() {
       }
     }
 
+    sortRowsByTimeIfNeeded();

Review Comment:
   The double copy is gone: the generator no longer sorts, it only copies the 
batch. The reorder happens once on the coordinator and replaces the statement 
containers.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
integration-test/src/test/java/org/apache/iotdb/relational/it/insertquery/IoTDBInsertQueryIT.java:
##########
@@ -87,7 +87,11 @@ public static void setUpClass() {
         .getConfig()
         .getCommonConfig()
         .setPartitionInterval(1000)
-        .setMemtableSizeThreshold(10000);
+        .setMemtableSizeThreshold(10000)
+        // The out-of-order tablet test needs a sequence file with several 
pages: when the earlier
+        // file only has one page, the read side merges the overlapping page 
before the earlier
+        // pages are emitted and the wrong order is masked.
+        .setMaxNumberOfPointsInPage(10);

Review Comment:
   Removed the class-wide `setMaxNumberOfPointsInPage(10)`. The new test 
creates its own tables and asserts the row sequence, which depends on the 
overlapping sequence file rather than on the page layout, so the rest of the 
class keeps the default layout.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -423,13 +427,51 @@ public ExecutionResult executeForTreeModel(
     }
   }
 
+  /**
+   * The storage layer assumes the rows of a tablet are sorted by time: the 
sequence/unsequence
+   * classification, the time boundaries of the tsfile resource, the ttl check 
and the time
+   * partition routing all rely on it. The child operator of an {@code INSERT 
INTO ... SELECT} does
+   * not guarantee that order, and REST/MQTT build the statement from the 
request as received, so
+   * restore it here: this runs on the coordinator, before the tablet is 
routed and before the write
+   * lock is taken. A tablet that is already sorted only costs one pass over 
its timestamps.
+   */
+  private static void sortTabletStatementIfNecessary(final Statement 
statement) {
+    // The pipe receiver and the write-back sink wrap the statement before 
submitting it.
+    final Statement target =
+        statement instanceof PipeEnrichedStatement
+            ? ((PipeEnrichedStatement) statement).getInnerStatement()
+            : statement;
+    if (target instanceof InsertTabletStatement) {
+      sortTabletStatement((InsertTabletStatement) target);
+    } else if (target instanceof InsertMultiTabletsStatement) {
+      for (final InsertTabletStatement tablet :
+          ((InsertMultiTabletsStatement) 
target).getInsertTabletStatementList()) {
+        sortTabletStatement(tablet);
+      }
+    }
+  }
+
+  private static void sortTabletStatement(final InsertTabletStatement tablet) {
+    if (tablet.getColumnCategories() == null) {
+      // A tree-model tablet holds a single device: sorting by time is all 
that is needed.
+      new InsertEventSorter(tablet).sortByTimestampIfNecessary();
+    } else {
+      // A table-model tablet can hold several devices. Keeping them together 
preserves the device
+      // segments the storage layer splits the tablet into, so only the times 
inside a device are
+      // reordered.
+      new 
TableModelTabletEventSorter(tablet).sortByDeviceIdAndTimestampIfNecessary();

Review Comment:
   Fixed. The coordinator records the permutation on the statement, the planner 
carries it to the node, and `splitByReplicaSet` now emits the redirect 
endpoints in the submitted row order, so the client row-to-device association 
stays correct.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -423,13 +427,51 @@ public ExecutionResult executeForTreeModel(
     }
   }
 
+  /**
+   * The storage layer assumes the rows of a tablet are sorted by time: the 
sequence/unsequence
+   * classification, the time boundaries of the tsfile resource, the ttl check 
and the time
+   * partition routing all rely on it. The child operator of an {@code INSERT 
INTO ... SELECT} does
+   * not guarantee that order, and REST/MQTT build the statement from the 
request as received, so
+   * restore it here: this runs on the coordinator, before the tablet is 
routed and before the write
+   * lock is taken. A tablet that is already sorted only costs one pass over 
its timestamps.
+   */
+  private static void sortTabletStatementIfNecessary(final Statement 
statement) {
+    // The pipe receiver and the write-back sink wrap the statement before 
submitting it.
+    final Statement target =
+        statement instanceof PipeEnrichedStatement
+            ? ((PipeEnrichedStatement) statement).getInnerStatement()
+            : statement;
+    if (target instanceof InsertTabletStatement) {
+      sortTabletStatement((InsertTabletStatement) target);
+    } else if (target instanceof InsertMultiTabletsStatement) {
+      for (final InsertTabletStatement tablet :
+          ((InsertMultiTabletsStatement) 
target).getInsertTabletStatementList()) {
+        sortTabletStatement(tablet);
+      }
+    }
+  }
+
+  private static void sortTabletStatement(final InsertTabletStatement tablet) {
+    if (tablet.getColumnCategories() == null) {
+      // A tree-model tablet holds a single device: sorting by time is all 
that is needed.
+      new InsertEventSorter(tablet).sortByTimestampIfNecessary();
+    } else {
+      // A table-model tablet can hold several devices. Keeping them together 
preserves the device
+      // segments the storage layer splits the tablet into, so only the times 
inside a device are
+      // reordered.
+      new 
TableModelTabletEventSorter(tablet).sortByDeviceIdAndTimestampIfNecessary();

Review Comment:
   Fixed. `InsertTabletStatementAdapter#detachContainers` copies the outer 
timestamp, column and bitmap containers right before a reorder, so a statement 
built from a node or a tablet never changes the source arrays.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
integration-test/src/test/java/org/apache/iotdb/relational/it/insertquery/IoTDBInsertQueryIT.java:
##########
@@ -167,6 +167,83 @@ public void testSimple() throws SQLException {
     }
   }
 
+  /**
+   * End-to-end check for an {@code INSERT INTO ... SELECT} whose source 
tablet is not in ascending
+   * time order. The destination device is flushed at time 39, then the query 
feeds it another
+   * tablet: on the unfixed code the row at time 15 is classified as sequence 
data, a sequence
+   * TsFile overlapping [10,39] is created, and the query returns 15 after 39 
(or twice).
+   *
+   * <p>The out-of-order input depends on the nested-loop cross join emitting 
the rows in probe
+   * order, i.e. {@code [100, 15]}. If the optimizer ever emits those rows in 
time order this test
+   * no longer reproduces the situation and would pass trivially; the guards 
that do not depend on
+   * the plan shape are {@code InsertTabletStatementGeneratorTest} (the tablet 
is sorted before it
+   * leaves the engine) and {@code
+   * DataRegionTest#testOutOfOrderTabletKeepsSequenceSpaceNonOverlapped} (the 
storage-layer
+   * classification of an unsorted tablet).
+   *
+   * <p>The first sequence file also has to hold several pages (see {@code
+   * max_number_of_points_in_page} in {@link #setUpClass()}): with a single 
page the overlapping
+   * page is merged before the earlier pages are emitted, which hides the 
wrong order as well.

Review Comment:
   The test no longer relies on a page layout: it creates its own tables, 
inserts through `INSERT INTO ... SELECT`, and asserts the exact row sequence. 
The failure it guards against is the overlapping sequence file, not a 
page-level merge.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -676,6 +718,7 @@ private IQueryExecution createQueryExecutionForTableModel(
       Metadata metadata,
       long timeOut,
       long startTime) {
+    sortTabletStatementIfNecessary(statement);

Review Comment:
   Fixed. `InsertTablet#getAttributeValueList` and the last-index map now 
iterate the rows in the submitted order using the recorded permutation, so the 
last submitted non-null value still wins.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to