luoluoyuyu opened a new pull request, #19313:
URL: https://github.com/apache/iotdb/pull/19313

   ## Problem
   
   When `tvlist_sort_threshold > 0`, an aligned memchunk is flushed through the 
single-`TVList` iterator path (`AlignedTVList.TVListIterator.encodeBatch`). 
That iterator stops as soon as the **chunk** point cap is reached and only 
writes the values of the current page into the value chunk writers; the 
matching time page is written by the caller (`AlignedWritableMemChunk.encode`) 
only when the **page** cap is reached too.
   
   `AlignedWritableMemChunk.encode` then sealed the chunk at the chunk cap 
without flushing that pending partial page. `TimeChunkWriter.sealCurrentPage()` 
writes nothing when the time page has no points, while 
`ValueChunkWriter.sealCurrentPage()` writes the pending values, so the time 
chunk ended up with one page less than the value chunks. 
`AbstractAlignedChunkReader.initAllPageReaders()` walks the time chunk buffer, 
so the rows of the extra value page were silently unreadable, and the 
time/value chunk statistics (`count`/`maxTime`) diverged.
   
   The effective chunk cap is
   
   ```text
   min(target_chunk_point_num, target_chunk_size / avg point size of the 
largest column)
   ```
   
   so it is usually not a multiple of the page cap (1000 by default) once a 
wide TEXT column narrows it, e.g. `262144 / ~330 = 789`. The chunk is then 
sealed in the middle of a page, so nearly every chunk of such a series is 
affected.
   
   The default path (`tvlist_sort_threshold = 0`, `encodeWorkingAlignedTVList`) 
writes every page explicitly, and 
`MergeSortMultiAlignedTVListIterator.encodeBatch` writes the page itself before 
breaking, so only the single-`TVList` path was broken.
   
   ## Reproduction
   
   2400 points, page cap 1000, chunk cap 1500, `tvlist_sort_threshold` above 
the row count:
   
   - Before: the first encoded aligned chunk contained **1000** points in its 
time chunk but **1500** in its value chunk (the new UT fails with 
`expected:<1000> but was:<1500>`).
   - End-to-end with a 300-char TEXT column (chunk cap ~789) and 3000 rows: 
before the fix, only the last 633 rows can be read back from the TsFile and the 
first 2367 rows are lost (the reader returns timestamp 2367 as the first row); 
after the fix all 3000 rows are read back with their values.
   
   ## Fix
   
   Flush the pending partial page before sealing in **both** places that seal a 
chunk, through a shared helper `flushPendingPageAndEnqueueChunk()`, so the 
chunk-cap branch and the last-batch branch cannot diverge again. 
`AlignedChunkWriterImpl.write()` followed by `sealCurrentPage()` now puts the 
time page and the value pages of the partial page into the same chunk.
   
   ## Tests
   
   - New UT 
`AlignedWritableMemChunkTest#testEncodeFlushesPendingPageBeforeSealingAtChunkCap`:
 asserts that the time chunk and every value chunk of one aligned chunk contain 
the same number of points, and that all rows are written.
   - New end-to-end UT 
`MemTableFlushTaskTest#testAlignedFlushKeepsAllRowsWhenChunkCapIsNotAPageMultiple`:
 flushes a wide-TEXT aligned memchunk to a TsFile and reads every row back.
   - Both tests were verified to fail before the fix and pass after it.
   - Regression run: `AlignedWritableMemChunkTest`, `MemTableFlushTaskTest`, 
`AlignedTVListIteratorTest`, `NonAlignedTVListIteratorTest`, 
`MemChunkDeserializeTest` — 55 tests, 0 failures.
   
   This only affects the non-default `tvlist_sort_threshold > 0` tuning; the 
default configuration is not affected.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to