ColinLeeo opened a new issue, #19020:
URL: https://github.com/apache/iotdb/issues/19020

   ### Summary
   
   A sequence (seq) TsFile can be created whose time range **overlaps an 
existing seq TsFile of the same series**. Once that state exists, 
`SeriesScanUtil` silently returns **duplicated and out-of-order rows** for a 
plain `SELECT` (no aggregation involved, no exception raised).
   
   The overlapping seq file is produced by the engine itself through `INSERT 
INTO ... SELECT`, so no manual file manipulation, no crash, and no abnormal 
configuration is required.
   
   ### Environment
   
   - Apache IoTDB `2.0.11-SNAPSHOT`, `master` @ `ba287a9cb4`
   - Single node, table model
   - `max_number_of_points_in_page=10` (only to create a multi-page chunk from 
a small data set; the default is `10000`, so in production the same scenario 
needs >10000 points in one seq file, which is trivially reached)
   
   ### Reproduction (pure SQL)
   
   ```sql
   CREATE DATABASE test;
   USE test;
   CREATE TABLE dst (id string tag, s int32 field);
   CREATE TABLE src (id string tag, s int32 field);
   
   -- (1) create seq file S1 covering [10,39]; 15 is intentionally omitted
   INSERT INTO dst (time, id, s) VALUES
    (10,'x',10),(11,'x',11),(12,'x',12),(13,'x',13),(14,'x',14),
    (16,'x',16),(17,'x',17),(18,'x',18),(19,'x',19),(20,'x',20),
    (21,'x',21),(22,'x',22),(23,'x',23),(24,'x',24),(25,'x',25),
    (26,'x',26),(27,'x',27),(28,'x',28),(29,'x',29),(30,'x',30),
    (31,'x',31),(32,'x',32),(33,'x',33),(34,'x',34),(35,'x',35),
    (36,'x',36),(37,'x',37),(38,'x',38),(39,'x',39);
   FLUSH;
   
   INSERT INTO src (time, id, s) VALUES (100,'x',1),(15,'x',2);
   
   -- (2) the child operator emits rows in nested-loop order [100, 15],
   --     which makes the tablet below out of time order
   INSERT INTO dst (SELECT b.time, b.id, b.s FROM src a, src b
                    WHERE a.id='x' AND b.id='x' AND a.time<>b.time);
   FLUSH;
   
   SELECT time, s FROM dst WHERE id='x';
   ```
   
   > Note: `ORDER BY` inside `INSERT INTO ... SELECT` is optimized away and 
`UNION ALL` as an insert source silently inserts 0 rows, so the join form above 
is used to obtain a deterministic, unsorted tablet. A fresh database is 
required, because `HashLastFlushTimeMap` may degrade to partition-level flush 
times and then a new device inherits `lastFlushTime` from other devices 
(everything goes to unseq and the issue does not reproduce).
   
   ### Expected
   
   31 rows, strictly increasing by time:
   
   ```
   10 11 12 13 14 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 
36 37 38 39 100
   ```
   
   ### Actual
   
   31 rows, but `15` is emitted **after** `39`:
   
   ```
   10 11 12 13 14 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 
36 37 38 39 15 100
   ```
   
   If the variant where `15` already exists in S1 (i.e. `(15,'x',15)` in step 1 
and `(15,'x',2)` in the `src` insert) is used, the query returns **both 
versions of timestamp 15** (`s=15` and `s=2`), i.e. de-duplication across files 
also breaks.
   
   On disk, the device's region then contains two seq TsFiles whose range for 
the same series overlaps (`[10,39]` and `[15,100]`).
   
   ### Root cause
   
   1. `INSERT INTO ... SELECT` builds a tablet in the child operator's output 
order without sorting:
      
[TableInsertTabletStatementGenerator.processTsBlock()](https://github.com/apache/iotdb/blob/master/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/TableInsertTabletStatementGenerator.java#L76-L102)
      copies rows verbatim. The storage layer, however, documents that it 
*assumes the batch has been sorted by the client* 
(`InsertTabletNode.checkTTLInternal`, "assume that batch has been sorted by 
client").
   
   2. `DataRegion.split()` latches `isSequence`:
      
[DataRegion.java#L1268-L1296](https://github.com/apache/iotdb/blob/master/iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/DataRegion.java#L1268-L1296)
      once a row is classified as sequence, every following row of the same 
device is treated as sequence as well, regardless of its timestamp. With the 
tablet `[100, 15]`, row `15` (which is older than the devic's last flush time) 
is therefore written into a **sequence** TsFile, producing an overlap.
   
   3. `SeriesScanUtil` assumes the seq page readers are globally ordered by 
time:
      `unpackAllOverlappedChunkMetadataToPageReaders()` appends the pages of 
every overlapping seq chunk to the append-only `seqPageReaders` list, while 
`getFirstPageReaderFromCachedReaders()` / `firstPageOverlapped()` only ever 
look at `seqPageReaders.get(0)`. The page of the second (overlapping) seq file 
therefore stays invisible until the first file is exhausted, so the overlapping 
range is neither merged nor de-duplicated — the merged reader never sees it.
   
   ### Suggested fix (in priority order)
   
   1. **`DataRegion.split()`**: drop the `isSequence` latch and re-evaluate 
`time > lastFlushTime` per row, or sort the tablet by `(device, time)` before 
splitting. This removes the root cause and also closes every other ingress that 
can produce overlapping seq files.
   2. **`TableIntoOperator`**: sort rows by `(device, time)` before building 
the insert tablet, so the documented "sorted by client" precondition holds for 
the engine's own SQL path.
   3. **`SeriesScanUtil`** (defensive): pick the page reader with the smallest 
start time instead of `seqPageReaders.get(0)` (or at least assert the ordering 
invariant), so that any pre-existing overlapping state cannot silently produce 
duplicated/out-of-order rows.
   
   ### Impact
   
   Silent wrong query results (duplicated rows and broken time order) on a 
plain `SELECT`; no error, no warning in the log. Any client that assumes `ORDER 
BY time`-equivalent ordering of a raw series scan will produce incorrect 
downstream results.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to