ColinLeeo commented on code in PR #19035:
URL: https://github.com/apache/iotdb/pull/19035#discussion_r4463634378
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/InsertTabletStatementGenerator.java:
##########
@@ -157,6 +161,64 @@ protected void processColumn(
.write(sourceTypeConvertor, valueColumn, rowIndex, columns, rowCount);
}
+ /**
+ * The storage layer assumes the rows of a tablet are sorted by time: the
sequence/unsequence
+ * classification, the time boundaries of the tsfile resource, the ttl
check, the time partition
+ * routing and the last value composition all rely on it. The child operator
of an {@code INSERT
+ * INTO ... SELECT} does not guarantee that order, so restore it before the
tablet leaves the
+ * engine.
+ */
+ private void sortRowsByTimeIfNeeded() {
+ boolean sorted = true;
+ for (int i = 1; i < rowCount; i++) {
+ if (times[i] < times[i - 1]) {
+ sorted = false;
+ break;
+ }
+ }
+ if (sorted) {
+ return;
+ }
+
+ Integer[] order = new Integer[rowCount];
+ for (int i = 0; i < rowCount; i++) {
+ order[i] = i;
+ }
+ // stable, so rows sharing a timestamp keep their relative order
+ Arrays.sort(order, Comparator.comparingLong(index -> times[index]));
+
+ int[] permutation = new int[rowCount];
+ long[] sortedTimes = new long[rowCount];
+ for (int i = 0; i < rowCount; i++) {
+ permutation[i] = order[i];
+ sortedTimes[i] = times[order[i]];
+ }
+ times = sortedTimes;
+ for (int i = 0; i < columns.length; i++) {
+ columns[i] = permuteColumn(columns[i], permutation);
+ bitMaps[i] = permuteBitMap(bitMaps[i], permutation);
+ }
+ }
+
+ private Object permuteColumn(Object column, int[] permutation) {
+ int length = Array.getLength(column);
+ Object sortedColumn =
Array.newInstance(column.getClass().getComponentType(), length);
+ for (int i = 0; i < length; i++) {
+ Array.set(sortedColumn, i, Array.get(column, permutation[i]));
Review Comment:
Replaced. The tablet is now sorted through the typed reorder of
`PipeInsertEventSorter` (`reorderValueListAndBitMap`, which switches on the
column type and copies with typed loops); the reflective
`Array.get`/`Array.set` copy is gone.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/InsertTabletStatementGenerator.java:
##########
@@ -157,6 +161,64 @@ protected void processColumn(
.write(sourceTypeConvertor, valueColumn, rowIndex, columns, rowCount);
}
+ /**
+ * The storage layer assumes the rows of a tablet are sorted by time: the
sequence/unsequence
+ * classification, the time boundaries of the tsfile resource, the ttl
check, the time partition
+ * routing and the last value composition all rely on it. The child operator
of an {@code INSERT
+ * INTO ... SELECT} does not guarantee that order, so restore it before the
tablet leaves the
+ * engine.
+ */
+ private void sortRowsByTimeIfNeeded() {
+ boolean sorted = true;
+ for (int i = 1; i < rowCount; i++) {
+ if (times[i] < times[i - 1]) {
+ sorted = false;
+ break;
+ }
+ }
+ if (sorted) {
+ return;
+ }
+
+ Integer[] order = new Integer[rowCount];
+ for (int i = 0; i < rowCount; i++) {
+ order[i] = i;
+ }
+ // stable, so rows sharing a timestamp keep their relative order
+ Arrays.sort(order, Comparator.comparingLong(index -> times[index]));
Review Comment:
Fixed as suggested: a table-model tablet is now sorted by device and then by
time, so the device segments stay as few as the child operator produced them. A
tablet that is already in that order is left untouched after one pass over its
timestamps.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java:
##########
@@ -1368,17 +1379,17 @@ private void split(
before = loc;
beforeTimePartition = timePartitionId;
isSequence = time > lastFlushTime;
- } else if (!isSequence) {
+ } else if (!isSequence || !timeOrdered) {
Review Comment:
The storage-layer change is gone. The sort now happens in
`Coordinator#sortTabletStatementIfNecessary` for every tablet statement
(`INSERT INTO`, REST, MQTT, the session API and pipe), before routing and
outside the write lock, so the storage layer keeps assuming sorted input as
documented.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/InsertTabletStatementGenerator.java:
##########
@@ -157,6 +161,64 @@ protected void processColumn(
.write(sourceTypeConvertor, valueColumn, rowIndex, columns, rowCount);
}
+ /**
+ * The storage layer assumes the rows of a tablet are sorted by time: the
sequence/unsequence
+ * classification, the time boundaries of the tsfile resource, the ttl
check, the time partition
+ * routing and the last value composition all rely on it. The child operator
of an {@code INSERT
+ * INTO ... SELECT} does not guarantee that order, so restore it before the
tablet leaves the
+ * engine.
+ */
+ private void sortRowsByTimeIfNeeded() {
Review Comment:
Done. The sorter moved to `org.apache.iotdb.db.utils.sorter` (the `Pipe`
prefix is dropped) and the query engine reuses it; the pipe classes only follow
the new package.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java:
##########
@@ -1368,17 +1379,17 @@ private void split(
before = loc;
beforeTimePartition = timePartitionId;
isSequence = time > lastFlushTime;
- } else if (!isSequence) {
+ } else if (!isSequence || !timeOrdered) {
lastFlushTime = getLastFlushTime(timePartitionId,
insertTabletNode.getDeviceID(loc));
- if (time > lastFlushTime) {
- // the same partition and switch to sequence data
- // insert previous range into unsequence
- updateSplitInfo(splitInfo, beforeTimePartition, false, new int[]
{before, loc});
+ boolean rowIsSequence = time > lastFlushTime;
Review Comment:
Not applicable anymore: the `DataRegion.split` change was removed, so the
write lock keeps its current logic.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]