luoluoyuyu opened a new pull request, #19341: URL: https://github.com/apache/iotdb/pull/19341
## Problem Fast compaction can copy value pages whose data has already been deleted into the target file, so the deleted data becomes visible again after compaction. For an aligned device, `AlignedSeriesBatchCompactionUtils#calculateAlignedPageModifiedStatus` aggregates the modified status of the value pages that form one aligned page. When one of the value pages is deleted completely while another value page of the same aligned page is not deleted at all, the aggregated status was `NONE_DELETED`, so the page was flushed to the target file directly by `AbstractCompactionWriter#flushAlignedPageToChunkWriter` (the raw page was copied as is). Deletions are materialized while compaction writes the target file, and the target file does not inherit the historical `.mods` of the source files (only the `.compaction.mods` are combined), so the deleted data of that value page is copied into the target file and revived. The wrong status was also recorded in `BatchCompactionPlan` and reused by the following column batches of a batched compaction. ### Reproduction Tree model, aligned device `root.sg.d` with `s0`, `s1`, `s2`, one chunk with two pages `[1, 1000]` and `[2001, 3000]`: ```sql DELETE FROM root.sg.d.s1 WHERE time <= 1000 ``` The deletion covers the first page of `s1` completely but not the whole chunk, so the chunk metadata of `s1` is kept and the first value page of `s1` is `ALL_DELETED`, while the first value pages of `s0` and `s2` are `NONE_DELETED`. * Before: after compaction, `s1` values in `[1, 1000]` are readable again from the compacted file. * After: the first value page of `s1` is written as an empty page, `s1` values in `[1, 1000]` stay deleted and `[2001, 3000]` are kept. ### Trigger conditions * the fast performer is used: cross space compaction, inner unseq compaction, or inner seq compaction configured with `FAST`; * a `DELETE` on a subset of the measurements of an aligned device (TTL is not affected, and neither is the case where a whole chunk of that measurement is deleted); * the deletion covers a whole page of that measurement but not the whole chunk; * the page is actually flushed directly, which requires the page to be large enough (by default `>= 100` points or `>= 1024` bytes). Smaller pages are deserialized and were not affected. The read chunk / read point performers are not affected: they already write an empty value page for a page that has been deleted completely. ## Fix * `AbstractCompactionWriter#flushAlignedPageToChunkWriter` now writes an empty value page for a value page whose data has been deleted completely (checked with the delete intervals of the value chunk metadata and the time range of the aligned page) instead of copying the raw page. This keeps both the page layout of the aligned chunk and the fast path. * Rewrite `AlignedSeriesBatchCompactionUtils#calculateAlignedPageModifiedStatus` to state the contract explicitly: `PARTIAL_DELETED` only if some value page is deleted partially, `ALL_DELETED` only if all the existing value pages are deleted completely, otherwise `NONE_DELETED` (the page may be flushed directly because the deleted value pages are written as empty pages). It no longer returns `null`. * Add the regression test `FastAlignedInnerCompactionWithDeletedValuePageTest`, which deletes a page-aligned time range on a subset of the value columns (1000 points per page) for both the single batch and the column-batched fast inner compaction. ## Tests * The new test fails on the old code with `expected:<null> but was:<1>` (the deleted value of `s1` at time 1 is revived) and passes with the fix. * `FastAlignedCrossCompactionTest`, `BatchedAlignedSeriesFastInnerCompactionTest`, `BatchedAlignedSeriesCrossSpaceCompactionTest`, `BatchedAlignedSeriesReadChunkCompactionTest`, `NewReadChunkCompactionPerformerWithAlignedSeriesTest`, `FastCompactionPerformerAlignedTest`, `FastCompactionPerformerWithEmptyPageTest`, `CompactionWithAllNullRowsTest`, `TableModelFastCompactionPerformerTest` and `FastNonAlignedCrossCompactionTest` pass. * `spotless:check` passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
