shuwenwei commented on PR #17941: URL: https://github.com/apache/iotdb/pull/17941#issuecomment-4706390784
I found one path where the new size checkpoint is updated but never consumed. `AbstractCompactionWriter` now increments `writtenPointTotalSizeArray` from direct page flushes, and `checkChunkSizeAndMayOpenANewChunk` can use that size checkpoint to force a chunk-size check. However, `FastInnerCompactionWriter` returns immediately after these direct page flushes: - `flushAlignedPage` after `flushAlignedPageToChunkWriter(...)` - `flushBatchedValuePage` after `flushAlignedPageToChunkWriter(...)` - `flushNonAlignedPage` after `flushNonAlignedPageToChunkWriter(...)` Unlike `FastCrossCompactionWriter`, these methods do not call `checkChunkSizeAndMayOpenANewChunk(...)` after the page is appended. For inner fast compaction with large TEXT/STRING/BLOB/OBJECT pages, the new `writtenPointTotalSizeArray` can cross the checkpoint, but no chunk-size check runs, so the writer may still keep accumulating pages until a later chunk boundary or `endMeasurement`. Could you call `checkChunkSizeAndMayOpenANewChunk(fileWriter, chunkWriters[subTaskId], subTaskId)` after each successful direct page flush in `FastInnerCompactionWriter`, and add a test that covers the inner-fast page flush path? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
