ColinLeeo commented on code in PR #19035:
URL: https://github.com/apache/iotdb/pull/19035#discussion_r4463635832
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/InsertTabletStatementGenerator.java:
##########
@@ -112,6 +114,8 @@ public InsertTabletStatement
constructInsertTabletStatement() {
}
}
+ sortRowsByTimeIfNeeded();
Review Comment:
The double copy is gone: the generator no longer sorts, it only copies the
batch. The reorder happens once on the coordinator and replaces the statement
containers.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/insertquery/IoTDBInsertQueryIT.java:
##########
@@ -87,7 +87,11 @@ public static void setUpClass() {
.getConfig()
.getCommonConfig()
.setPartitionInterval(1000)
- .setMemtableSizeThreshold(10000);
+ .setMemtableSizeThreshold(10000)
+ // The out-of-order tablet test needs a sequence file with several
pages: when the earlier
+ // file only has one page, the read side merges the overlapping page
before the earlier
+ // pages are emitted and the wrong order is masked.
+ .setMaxNumberOfPointsInPage(10);
Review Comment:
Removed the class-wide `setMaxNumberOfPointsInPage(10)`. The new test
creates its own tables and asserts the row sequence, which depends on the
overlapping sequence file rather than on the page layout, so the rest of the
class keeps the default layout.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -423,13 +427,51 @@ public ExecutionResult executeForTreeModel(
}
}
+ /**
+ * The storage layer assumes the rows of a tablet are sorted by time: the
sequence/unsequence
+ * classification, the time boundaries of the tsfile resource, the ttl check
and the time
+ * partition routing all rely on it. The child operator of an {@code INSERT
INTO ... SELECT} does
+ * not guarantee that order, and REST/MQTT build the statement from the
request as received, so
+ * restore it here: this runs on the coordinator, before the tablet is
routed and before the write
+ * lock is taken. A tablet that is already sorted only costs one pass over
its timestamps.
+ */
+ private static void sortTabletStatementIfNecessary(final Statement
statement) {
+ // The pipe receiver and the write-back sink wrap the statement before
submitting it.
+ final Statement target =
+ statement instanceof PipeEnrichedStatement
+ ? ((PipeEnrichedStatement) statement).getInnerStatement()
+ : statement;
+ if (target instanceof InsertTabletStatement) {
+ sortTabletStatement((InsertTabletStatement) target);
+ } else if (target instanceof InsertMultiTabletsStatement) {
+ for (final InsertTabletStatement tablet :
+ ((InsertMultiTabletsStatement)
target).getInsertTabletStatementList()) {
+ sortTabletStatement(tablet);
+ }
+ }
+ }
+
+ private static void sortTabletStatement(final InsertTabletStatement tablet) {
+ if (tablet.getColumnCategories() == null) {
+ // A tree-model tablet holds a single device: sorting by time is all
that is needed.
+ new InsertEventSorter(tablet).sortByTimestampIfNecessary();
+ } else {
+ // A table-model tablet can hold several devices. Keeping them together
preserves the device
+ // segments the storage layer splits the tablet into, so only the times
inside a device are
+ // reordered.
+ new
TableModelTabletEventSorter(tablet).sortByDeviceIdAndTimestampIfNecessary();
Review Comment:
Fixed. The coordinator records the permutation on the statement, the planner
carries it to the node, and `splitByReplicaSet` now emits the redirect
endpoints in the submitted row order, so the client row-to-device association
stays correct.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -423,13 +427,51 @@ public ExecutionResult executeForTreeModel(
}
}
+ /**
+ * The storage layer assumes the rows of a tablet are sorted by time: the
sequence/unsequence
+ * classification, the time boundaries of the tsfile resource, the ttl check
and the time
+ * partition routing all rely on it. The child operator of an {@code INSERT
INTO ... SELECT} does
+ * not guarantee that order, and REST/MQTT build the statement from the
request as received, so
+ * restore it here: this runs on the coordinator, before the tablet is
routed and before the write
+ * lock is taken. A tablet that is already sorted only costs one pass over
its timestamps.
+ */
+ private static void sortTabletStatementIfNecessary(final Statement
statement) {
+ // The pipe receiver and the write-back sink wrap the statement before
submitting it.
+ final Statement target =
+ statement instanceof PipeEnrichedStatement
+ ? ((PipeEnrichedStatement) statement).getInnerStatement()
+ : statement;
+ if (target instanceof InsertTabletStatement) {
+ sortTabletStatement((InsertTabletStatement) target);
+ } else if (target instanceof InsertMultiTabletsStatement) {
+ for (final InsertTabletStatement tablet :
+ ((InsertMultiTabletsStatement)
target).getInsertTabletStatementList()) {
+ sortTabletStatement(tablet);
+ }
+ }
+ }
+
+ private static void sortTabletStatement(final InsertTabletStatement tablet) {
+ if (tablet.getColumnCategories() == null) {
+ // A tree-model tablet holds a single device: sorting by time is all
that is needed.
+ new InsertEventSorter(tablet).sortByTimestampIfNecessary();
+ } else {
+ // A table-model tablet can hold several devices. Keeping them together
preserves the device
+ // segments the storage layer splits the tablet into, so only the times
inside a device are
+ // reordered.
+ new
TableModelTabletEventSorter(tablet).sortByDeviceIdAndTimestampIfNecessary();
Review Comment:
Fixed. `InsertTabletStatementAdapter#detachContainers` copies the outer
timestamp, column and bitmap containers right before a reorder, so a statement
built from a node or a tablet never changes the source arrays.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/insertquery/IoTDBInsertQueryIT.java:
##########
@@ -167,6 +167,83 @@ public void testSimple() throws SQLException {
}
}
+ /**
+ * End-to-end check for an {@code INSERT INTO ... SELECT} whose source
tablet is not in ascending
+ * time order. The destination device is flushed at time 39, then the query
feeds it another
+ * tablet: on the unfixed code the row at time 15 is classified as sequence
data, a sequence
+ * TsFile overlapping [10,39] is created, and the query returns 15 after 39
(or twice).
+ *
+ * <p>The out-of-order input depends on the nested-loop cross join emitting
the rows in probe
+ * order, i.e. {@code [100, 15]}. If the optimizer ever emits those rows in
time order this test
+ * no longer reproduces the situation and would pass trivially; the guards
that do not depend on
+ * the plan shape are {@code InsertTabletStatementGeneratorTest} (the tablet
is sorted before it
+ * leaves the engine) and {@code
+ * DataRegionTest#testOutOfOrderTabletKeepsSequenceSpaceNonOverlapped} (the
storage-layer
+ * classification of an unsorted tablet).
+ *
+ * <p>The first sequence file also has to hold several pages (see {@code
+ * max_number_of_points_in_page} in {@link #setUpClass()}): with a single
page the overlapping
+ * page is merged before the earlier pages are emitted, which hides the
wrong order as well.
Review Comment:
The test no longer relies on a page layout: it creates its own tables,
inserts through `INSERT INTO ... SELECT`, and asserts the exact row sequence.
The failure it guards against is the overlapping sequence file, not a
page-level merge.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -676,6 +718,7 @@ private IQueryExecution createQueryExecutionForTableModel(
Metadata metadata,
long timeOut,
long startTime) {
+ sortTabletStatementIfNecessary(statement);
Review Comment:
Fixed. `InsertTablet#getAttributeValueList` and the last-index map now
iterate the rows in the submitted order using the recorded permutation, so the
last submitted non-null value still wins.
_🤖 Addressed by [Claude Code](https://claude.com/claude-code)_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]