ColinLeeo opened a new pull request, #19035:
URL: https://github.com/apache/iotdb/pull/19035
## Problem
Fixes #19020.
A tablet whose rows are not in ascending time order could write a row
**older** than the device's last flush time into the **sequence** space. The
resulting sequence TsFile overlapped the existing sequence TsFile of the same
device, and plain `SELECT` queries then returned **duplicated rows and broke
the time order**, silently — no exception, no log entry.
A full pure-SQL reproduction (create the layout, insert an out-of-order
tablet through `INSERT INTO ... SELECT`, then observe `32` rows instead of `31`
with a duplicated timestamp out of place) is in the issue.
The state is reachable through the engine's own SQL path:
`TableInsertTabletStatementGenerator` builds the insert tablet in the child
operator's output order and does not sort it, while the storage layer documents
that it *assumes the batch has been sorted by the client*
(`InsertTabletNode.checkTTLInternal`).
## Root cause
`DataRegion.split()` classified the rows of a tablet by latching the
sequence flag:
```java
} else if (!isSequence) {
if (time > lastFlushTime) {
...
isSequence = true;
}
}
// else: isSequence not changed, just move the cursor forward
```
Once `isSequence` became `true`, every remaining row of the device was
classified as sequence regardless of its timestamp. For the tablet `[100, 15]`
of a device whose last flushed time is `39`, the row `15` was written into a
new sequence TsFile, producing an overlapping sequence file.
## Fix
Track whether the device's rows are still in ascending time order:
- **Sorted tablets (the normal case)** keep exactly the previous behaviour.
The only extra cost is one `time < previousTime` comparison per row; no
additional `lastFlushTime` lookups.
- **Once a descending step is seen**, each remaining row is classified
individually against the device's last flush time, and the range is split
whenever the classification changes. The consumer already supports several
ranges per partition.
Alternatives considered and not taken:
- *Sorting the tablet before the insert* — would require remapping the
`results` array and the `processedRows` bitset indices, a much larger change.
- *Sorting only in `TableIntoOperator`* — would close the SQL ingress only;
the session `insertTablet` API would still be able to feed unsorted rows.
Fixing it in the storage layer covers every ingress.
## Testing
New regression test
`DataRegionTest#testOutOfOrderTabletKeepsSequenceSpaceNonOverlapped`:
1. insert `10..39` for a device and close the file (last flushed time =
`39`);
2. insert the out-of-order tablet `[100, 15]` for the same device;
3. assert that the sequence space has no overlap
(`RepairDataFileScanUtil.checkTimePartitionHasOverlap`) and that `15` landed in
the unsequence space.
Verified that the test **fails on the unpatched code** (`unseqFiles`
expected `1`, was `0` — the older row went to the sequence space) and passes
with the patch. The full `DataRegionTest` suite also passes.
## Out of scope
- `SeriesScanUtil` still assumes the sequence page readers are globally
ordered. A defensive fix there (pick the reader with the smallest start time
instead of `seqPageReaders.get(0)`) protects against already-overlapping
states, but it changes the read/merge path and deserves its own PR with broader
regression testing.
- Pre-sorting in `TableIntoOperator` is a possible performance follow-up
(fewer ranges per tablet); it is not required for correctness.
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]