ColinLeeo opened a new pull request, #19035:
URL: https://github.com/apache/iotdb/pull/19035

   ## Problem
   
   Fixes #19020.
   
   A tablet whose rows are not in ascending time order could write a row 
**older** than the device's last flush time into the **sequence** space. The 
resulting sequence TsFile overlapped the existing sequence TsFile of the same 
device, and plain `SELECT` queries then returned **duplicated rows and broke 
the time order**, silently — no exception, no log entry.
   
   A full pure-SQL reproduction (create the layout, insert an out-of-order 
tablet through `INSERT INTO ... SELECT`, then observe `32` rows instead of `31` 
with a duplicated timestamp out of place) is in the issue.
   
   The state is reachable through the engine's own SQL path: 
`TableInsertTabletStatementGenerator` builds the insert tablet in the child 
operator's output order and does not sort it, while the storage layer documents 
that it *assumes the batch has been sorted by the client* 
(`InsertTabletNode.checkTTLInternal`).
   
   ## Root cause
   
   `DataRegion.split()` classified the rows of a tablet by latching the 
sequence flag:
   
   ```java
   } else if (!isSequence) {
     if (time > lastFlushTime) {
       ...
       isSequence = true;
     }
   }
   // else: isSequence not changed, just move the cursor forward
   ```
   
   Once `isSequence` became `true`, every remaining row of the device was 
classified as sequence regardless of its timestamp. For the tablet `[100, 15]` 
of a device whose last flushed time is `39`, the row `15` was written into a 
new sequence TsFile, producing an overlapping sequence file.
   
   ## Fix
   
   Track whether the device's rows are still in ascending time order:
   
   - **Sorted tablets (the normal case)** keep exactly the previous behaviour. 
The only extra cost is one `time < previousTime` comparison per row; no 
additional `lastFlushTime` lookups.
   - **Once a descending step is seen**, each remaining row is classified 
individually against the device's last flush time, and the range is split 
whenever the classification changes. The consumer already supports several 
ranges per partition.
   
   Alternatives considered and not taken:
   
   - *Sorting the tablet before the insert* — would require remapping the 
`results` array and the `processedRows` bitset indices, a much larger change.
   - *Sorting only in `TableIntoOperator`* — would close the SQL ingress only; 
the session `insertTablet` API would still be able to feed unsorted rows. 
Fixing it in the storage layer covers every ingress.
   
   ## Testing
   
   New regression test 
`DataRegionTest#testOutOfOrderTabletKeepsSequenceSpaceNonOverlapped`:
   
   1. insert `10..39` for a device and close the file (last flushed time = 
`39`);
   2. insert the out-of-order tablet `[100, 15]` for the same device;
   3. assert that the sequence space has no overlap 
(`RepairDataFileScanUtil.checkTimePartitionHasOverlap`) and that `15` landed in 
the unsequence space.
   
   Verified that the test **fails on the unpatched code** (`unseqFiles` 
expected `1`, was `0` — the older row went to the sequence space) and passes 
with the patch. The full `DataRegionTest` suite also passes.
   
   ## Out of scope
   
   - `SeriesScanUtil` still assumes the sequence page readers are globally 
ordered. A defensive fix there (pick the reader with the smallest start time 
instead of `seqPageReaders.get(0)`) protects against already-overlapping 
states, but it changes the read/merge path and deserves its own PR with broader 
regression testing.
   - Pre-sorting in `TableIntoOperator` is a possible performance follow-up 
(fewer ranges per tablet); it is not required for correctness.
   
   ---
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to