ColinLeeo commented on code in PR #19035:
URL: https://github.com/apache/iotdb/pull/19035#discussion_r4463634378


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/InsertTabletStatementGenerator.java:
##########
@@ -157,6 +161,64 @@ protected void processColumn(
         .write(sourceTypeConvertor, valueColumn, rowIndex, columns, rowCount);
   }
 
+  /**
+   * The storage layer assumes the rows of a tablet are sorted by time: the 
sequence/unsequence
+   * classification, the time boundaries of the tsfile resource, the ttl 
check, the time partition
+   * routing and the last value composition all rely on it. The child operator 
of an {@code INSERT
+   * INTO ... SELECT} does not guarantee that order, so restore it before the 
tablet leaves the
+   * engine.
+   */
+  private void sortRowsByTimeIfNeeded() {
+    boolean sorted = true;
+    for (int i = 1; i < rowCount; i++) {
+      if (times[i] < times[i - 1]) {
+        sorted = false;
+        break;
+      }
+    }
+    if (sorted) {
+      return;
+    }
+
+    Integer[] order = new Integer[rowCount];
+    for (int i = 0; i < rowCount; i++) {
+      order[i] = i;
+    }
+    // stable, so rows sharing a timestamp keep their relative order
+    Arrays.sort(order, Comparator.comparingLong(index -> times[index]));
+
+    int[] permutation = new int[rowCount];
+    long[] sortedTimes = new long[rowCount];
+    for (int i = 0; i < rowCount; i++) {
+      permutation[i] = order[i];
+      sortedTimes[i] = times[order[i]];
+    }
+    times = sortedTimes;
+    for (int i = 0; i < columns.length; i++) {
+      columns[i] = permuteColumn(columns[i], permutation);
+      bitMaps[i] = permuteBitMap(bitMaps[i], permutation);
+    }
+  }
+
+  private Object permuteColumn(Object column, int[] permutation) {
+    int length = Array.getLength(column);
+    Object sortedColumn = 
Array.newInstance(column.getClass().getComponentType(), length);
+    for (int i = 0; i < length; i++) {
+      Array.set(sortedColumn, i, Array.get(column, permutation[i]));

Review Comment:
   Replaced. The tablet is now sorted through the typed reorder of 
`PipeInsertEventSorter` (`reorderValueListAndBitMap`, which switches on the 
column type and copies with typed loops); the reflective 
`Array.get`/`Array.set` copy is gone.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/InsertTabletStatementGenerator.java:
##########
@@ -157,6 +161,64 @@ protected void processColumn(
         .write(sourceTypeConvertor, valueColumn, rowIndex, columns, rowCount);
   }
 
+  /**
+   * The storage layer assumes the rows of a tablet are sorted by time: the 
sequence/unsequence
+   * classification, the time boundaries of the tsfile resource, the ttl 
check, the time partition
+   * routing and the last value composition all rely on it. The child operator 
of an {@code INSERT
+   * INTO ... SELECT} does not guarantee that order, so restore it before the 
tablet leaves the
+   * engine.
+   */
+  private void sortRowsByTimeIfNeeded() {
+    boolean sorted = true;
+    for (int i = 1; i < rowCount; i++) {
+      if (times[i] < times[i - 1]) {
+        sorted = false;
+        break;
+      }
+    }
+    if (sorted) {
+      return;
+    }
+
+    Integer[] order = new Integer[rowCount];
+    for (int i = 0; i < rowCount; i++) {
+      order[i] = i;
+    }
+    // stable, so rows sharing a timestamp keep their relative order
+    Arrays.sort(order, Comparator.comparingLong(index -> times[index]));

Review Comment:
   Fixed as suggested: a table-model tablet is now sorted by device and then by 
time, so the device segments stay as few as the child operator produced them. A 
tablet that is already in that order is left untouched after one pass over its 
timestamps.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java:
##########
@@ -1368,17 +1379,17 @@ private void split(
         before = loc;
         beforeTimePartition = timePartitionId;
         isSequence = time > lastFlushTime;
-      } else if (!isSequence) {
+      } else if (!isSequence || !timeOrdered) {

Review Comment:
   The storage-layer change is gone. The sort now happens in 
`Coordinator#sortTabletStatementIfNecessary` for every tablet statement 
(`INSERT INTO`, REST, MQTT, the session API and pipe), before routing and 
outside the write lock, so the storage layer keeps assuming sorted input as 
documented.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/InsertTabletStatementGenerator.java:
##########
@@ -157,6 +161,64 @@ protected void processColumn(
         .write(sourceTypeConvertor, valueColumn, rowIndex, columns, rowCount);
   }
 
+  /**
+   * The storage layer assumes the rows of a tablet are sorted by time: the 
sequence/unsequence
+   * classification, the time boundaries of the tsfile resource, the ttl 
check, the time partition
+   * routing and the last value composition all rely on it. The child operator 
of an {@code INSERT
+   * INTO ... SELECT} does not guarantee that order, so restore it before the 
tablet leaves the
+   * engine.
+   */
+  private void sortRowsByTimeIfNeeded() {

Review Comment:
   Done. The sorter moved to `org.apache.iotdb.db.utils.sorter` (the `Pipe` 
prefix is dropped) and the query engine reuses it; the pipe classes only follow 
the new package.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java:
##########
@@ -1368,17 +1379,17 @@ private void split(
         before = loc;
         beforeTimePartition = timePartitionId;
         isSequence = time > lastFlushTime;
-      } else if (!isSequence) {
+      } else if (!isSequence || !timeOrdered) {
         lastFlushTime = getLastFlushTime(timePartitionId, 
insertTabletNode.getDeviceID(loc));
-        if (time > lastFlushTime) {
-          // the same partition and switch to sequence data
-          // insert previous range into unsequence
-          updateSplitInfo(splitInfo, beforeTimePartition, false, new int[] 
{before, loc});
+        boolean rowIsSequence = time > lastFlushTime;

Review Comment:
   Not applicable anymore: the `DataRegion.split` change was removed, so the 
write lock keeps its current logic.
   
   _🤖 Addressed by [Claude Code](https://claude.com/claude-code)_



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to