luoluoyuyu opened a new pull request, #19313: URL: https://github.com/apache/iotdb/pull/19313
## Problem When `tvlist_sort_threshold > 0`, an aligned memchunk is flushed through the single-`TVList` iterator path (`AlignedTVList.TVListIterator.encodeBatch`). That iterator stops as soon as the **chunk** point cap is reached and only writes the values of the current page into the value chunk writers; the matching time page is written by the caller (`AlignedWritableMemChunk.encode`) only when the **page** cap is reached too. `AlignedWritableMemChunk.encode` then sealed the chunk at the chunk cap without flushing that pending partial page. `TimeChunkWriter.sealCurrentPage()` writes nothing when the time page has no points, while `ValueChunkWriter.sealCurrentPage()` writes the pending values, so the time chunk ended up with one page less than the value chunks. `AbstractAlignedChunkReader.initAllPageReaders()` walks the time chunk buffer, so the rows of the extra value page were silently unreadable, and the time/value chunk statistics (`count`/`maxTime`) diverged. The effective chunk cap is ```text min(target_chunk_point_num, target_chunk_size / avg point size of the largest column) ``` so it is usually not a multiple of the page cap (1000 by default) once a wide TEXT column narrows it, e.g. `262144 / ~330 = 789`. The chunk is then sealed in the middle of a page, so nearly every chunk of such a series is affected. The default path (`tvlist_sort_threshold = 0`, `encodeWorkingAlignedTVList`) writes every page explicitly, and `MergeSortMultiAlignedTVListIterator.encodeBatch` writes the page itself before breaking, so only the single-`TVList` path was broken. ## Reproduction 2400 points, page cap 1000, chunk cap 1500, `tvlist_sort_threshold` above the row count: - Before: the first encoded aligned chunk contained **1000** points in its time chunk but **1500** in its value chunk (the new UT fails with `expected:<1000> but was:<1500>`). - End-to-end with a 300-char TEXT column (chunk cap ~789) and 3000 rows: before the fix, only the last 633 rows can be read back from the TsFile and the first 2367 rows are lost (the reader returns timestamp 2367 as the first row); after the fix all 3000 rows are read back with their values. ## Fix Flush the pending partial page before sealing in **both** places that seal a chunk, through a shared helper `flushPendingPageAndEnqueueChunk()`, so the chunk-cap branch and the last-batch branch cannot diverge again. `AlignedChunkWriterImpl.write()` followed by `sealCurrentPage()` now puts the time page and the value pages of the partial page into the same chunk. ## Tests - New UT `AlignedWritableMemChunkTest#testEncodeFlushesPendingPageBeforeSealingAtChunkCap`: asserts that the time chunk and every value chunk of one aligned chunk contain the same number of points, and that all rows are written. - New end-to-end UT `MemTableFlushTaskTest#testAlignedFlushKeepsAllRowsWhenChunkCapIsNotAPageMultiple`: flushes a wide-TEXT aligned memchunk to a TsFile and reads every row back. - Both tests were verified to fail before the fix and pass after it. - Regression run: `AlignedWritableMemChunkTest`, `MemTableFlushTaskTest`, `AlignedTVListIteratorTest`, `NonAlignedTVListIteratorTest`, `MemChunkDeserializeTest` — 55 tests, 0 failures. This only affects the non-default `tvlist_sort_threshold > 0` tuning; the default configuration is not affected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
