Copilot commented on code in PR #19035:
URL: https://github.com/apache/iotdb/pull/19035#discussion_r4462672866
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -423,13 +427,51 @@ public ExecutionResult executeForTreeModel(
}
}
+ /**
+ * The storage layer assumes the rows of a tablet are sorted by time: the
sequence/unsequence
+ * classification, the time boundaries of the tsfile resource, the ttl check
and the time
+ * partition routing all rely on it. The child operator of an {@code INSERT
INTO ... SELECT} does
+ * not guarantee that order, and REST/MQTT build the statement from the
request as received, so
+ * restore it here: this runs on the coordinator, before the tablet is
routed and before the write
+ * lock is taken. A tablet that is already sorted only costs one pass over
its timestamps.
+ */
+ private static void sortTabletStatementIfNecessary(final Statement
statement) {
+ // The pipe receiver and the write-back sink wrap the statement before
submitting it.
+ final Statement target =
+ statement instanceof PipeEnrichedStatement
+ ? ((PipeEnrichedStatement) statement).getInnerStatement()
+ : statement;
+ if (target instanceof InsertTabletStatement) {
+ sortTabletStatement((InsertTabletStatement) target);
+ } else if (target instanceof InsertMultiTabletsStatement) {
+ for (final InsertTabletStatement tablet :
+ ((InsertMultiTabletsStatement)
target).getInsertTabletStatementList()) {
+ sortTabletStatement(tablet);
+ }
+ }
+ }
+
+ private static void sortTabletStatement(final InsertTabletStatement tablet) {
+ if (tablet.getColumnCategories() == null) {
+ // A tree-model tablet holds a single device: sorting by time is all
that is needed.
+ new InsertEventSorter(tablet).sortByTimestampIfNecessary();
+ } else {
+ // A table-model tablet can hold several devices. Keeping them together
preserves the device
+ // segments the storage layer splits the tablet into, so only the times
inside a device are
+ // reordered.
+ new
TableModelTabletEventSorter(tablet).sortByDeviceIdAndTimestampIfNecessary();
Review Comment:
Grouping a session tablet by device changes the row order used for RPC
redirection. `RelationalInsertTabletNode.splitByReplicaSet()` builds
`redirectNodeList` in this reordered order, but
`Session.insertRelationalTabletOnce()` and `insertRelationalTabletByGroup()`
associate endpoint i with device i in the original client tablet. For an
already time-sorted request containing devices [b, a] on different leaders, the
response now contains [leader(a), leader(b)], poisoning both client
leader-cache entries and misrouting subsequent writes. Preserve the permutation
and restore row-indexed response metadata to request order before returning it.
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/insertquery/IoTDBInsertQueryIT.java:
##########
@@ -167,6 +167,83 @@ public void testSimple() throws SQLException {
}
}
+ /**
+ * End-to-end check for an {@code INSERT INTO ... SELECT} whose source
tablet is not in ascending
+ * time order. The destination device is flushed at time 39, then the query
feeds it another
+ * tablet: on the unfixed code the row at time 15 is classified as sequence
data, a sequence
+ * TsFile overlapping [10,39] is created, and the query returns 15 after 39
(or twice).
+ *
+ * <p>The out-of-order input depends on the nested-loop cross join emitting
the rows in probe
+ * order, i.e. {@code [100, 15]}. If the optimizer ever emits those rows in
time order this test
+ * no longer reproduces the situation and would pass trivially; the guards
that do not depend on
+ * the plan shape are {@code InsertTabletStatementGeneratorTest} (the tablet
is sorted before it
+ * leaves the engine) and {@code
+ * DataRegionTest#testOutOfOrderTabletKeepsSequenceSpaceNonOverlapped} (the
storage-layer
+ * classification of an unsorted tablet).
+ *
+ * <p>The first sequence file also has to hold several pages (see {@code
+ * max_number_of_points_in_page} in {@link #setUpClass()}): with a single
page the overlapping
+ * page is merged before the earlier pages are emitted, which hides the
wrong order as well.
Review Comment:
The 30 baseline rows do not establish the multi-page layout needed to expose
this regression: `setUpClass()` never lowers the page-point limit mentioned
here. With a single page, the unfixed reader can merge the overlap before
emitting baseline data, so the assertions can pass without the sorting fix. Set
`setMaxNumberOfPointsInPage(10)` before cluster startup and verify that the
regression fails without sorting.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -423,13 +427,51 @@ public ExecutionResult executeForTreeModel(
}
}
+ /**
+ * The storage layer assumes the rows of a tablet are sorted by time: the
sequence/unsequence
+ * classification, the time boundaries of the tsfile resource, the ttl check
and the time
+ * partition routing all rely on it. The child operator of an {@code INSERT
INTO ... SELECT} does
+ * not guarantee that order, and REST/MQTT build the statement from the
request as received, so
+ * restore it here: this runs on the coordinator, before the tablet is
routed and before the write
+ * lock is taken. A tablet that is already sorted only costs one pass over
its timestamps.
+ */
+ private static void sortTabletStatementIfNecessary(final Statement
statement) {
+ // The pipe receiver and the write-back sink wrap the statement before
submitting it.
+ final Statement target =
+ statement instanceof PipeEnrichedStatement
+ ? ((PipeEnrichedStatement) statement).getInnerStatement()
+ : statement;
+ if (target instanceof InsertTabletStatement) {
+ sortTabletStatement((InsertTabletStatement) target);
+ } else if (target instanceof InsertMultiTabletsStatement) {
+ for (final InsertTabletStatement tablet :
+ ((InsertMultiTabletsStatement)
target).getInsertTabletStatementList()) {
+ sortTabletStatement(tablet);
+ }
+ }
+ }
+
+ private static void sortTabletStatement(final InsertTabletStatement tablet) {
+ if (tablet.getColumnCategories() == null) {
+ // A tree-model tablet holds a single device: sorting by time is all
that is needed.
+ new InsertEventSorter(tablet).sortByTimestampIfNecessary();
+ } else {
+ // A table-model tablet can hold several devices. Keeping them together
preserves the device
+ // segments the storage layer splits the tablet into, so only the times
inside a device are
+ // reordered.
+ new
TableModelTabletEventSorter(tablet).sortByDeviceIdAndTimestampIfNecessary();
Review Comment:
When a pipe-backed table statement needs reordering, its
`InsertTabletStatement(InsertTabletNode)` constructor shares the source node's
outer column and bitmap arrays. The sorter replaces timestamps only on the
statement but changes those shared containers, leaving the source node with
mismatched timestamp/value pairs that another pipe can replicate. Give the
statement private column and bitmap containers before applying the permutation.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/Coordinator.java:
##########
@@ -676,6 +718,7 @@ private IQueryExecution createQueryExecutionForTableModel(
Metadata metadata,
long timeOut,
long startTime) {
+ sortTabletStatementIfNecessary(statement);
Review Comment:
Sorting before device-schema validation changes which attribute update wins:
for one device with input rows `(100, attr='old'), (15, attr='new')`,
`InsertTablet.getAttributeValueList()` now selects 'old' instead of the last
non-null input value, 'new'. Attributes are device metadata, not timestamped
fields. Preserve their original-order aggregation before sorting, for example
by sorting after device-schema validation removes attribute columns but before
partition routing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]