luoluoyuyu opened a new pull request, #19341:
URL: https://github.com/apache/iotdb/pull/19341

   ## Problem
   
   Fast compaction can copy value pages whose data has already been deleted 
into the target file, so the deleted data becomes visible again after 
compaction.
   
   For an aligned device, 
`AlignedSeriesBatchCompactionUtils#calculateAlignedPageModifiedStatus` 
aggregates the modified status of the value pages that form one aligned page. 
When one of the value pages is deleted completely while another value page of 
the same aligned page is not deleted at all, the aggregated status was 
`NONE_DELETED`, so the page was flushed to the target file directly by 
`AbstractCompactionWriter#flushAlignedPageToChunkWriter` (the raw page was 
copied as is).
   
   Deletions are materialized while compaction writes the target file, and the 
target file does not inherit the historical `.mods` of the source files (only 
the `.compaction.mods` are combined), so the deleted data of that value page is 
copied into the target file and revived. The wrong status was also recorded in 
`BatchCompactionPlan` and reused by the following column batches of a batched 
compaction.
   
   ### Reproduction
   
   Tree model, aligned device `root.sg.d` with `s0`, `s1`, `s2`, one chunk with 
two pages `[1, 1000]` and `[2001, 3000]`:
   
   ```sql
   DELETE FROM root.sg.d.s1 WHERE time <= 1000
   ```
   
   The deletion covers the first page of `s1` completely but not the whole 
chunk, so the chunk metadata of `s1` is kept and the first value page of `s1` 
is `ALL_DELETED`, while the first value pages of `s0` and `s2` are 
`NONE_DELETED`.
   
   * Before: after compaction, `s1` values in `[1, 1000]` are readable again 
from the compacted file.
   * After: the first value page of `s1` is written as an empty page, `s1` 
values in `[1, 1000]` stay deleted and `[2001, 3000]` are kept.
   
   ### Trigger conditions
   
   * the fast performer is used: cross space compaction, inner unseq 
compaction, or inner seq compaction configured with `FAST`;
   * a `DELETE` on a subset of the measurements of an aligned device (TTL is 
not affected, and neither is the case where a whole chunk of that measurement 
is deleted);
   * the deletion covers a whole page of that measurement but not the whole 
chunk;
   * the page is actually flushed directly, which requires the page to be large 
enough (by default `>= 100` points or `>= 1024` bytes). Smaller pages are 
deserialized and were not affected.
   
   The read chunk / read point performers are not affected: they already write 
an empty value page for a page that has been deleted completely.
   
   ## Fix
   
   * `AbstractCompactionWriter#flushAlignedPageToChunkWriter` now writes an 
empty value page for a value page whose data has been deleted completely 
(checked with the delete intervals of the value chunk metadata and the time 
range of the aligned page) instead of copying the raw page. This keeps both the 
page layout of the aligned chunk and the fast path.
   * Rewrite 
`AlignedSeriesBatchCompactionUtils#calculateAlignedPageModifiedStatus` to state 
the contract explicitly: `PARTIAL_DELETED` only if some value page is deleted 
partially, `ALL_DELETED` only if all the existing value pages are deleted 
completely, otherwise `NONE_DELETED` (the page may be flushed directly because 
the deleted value pages are written as empty pages). It no longer returns 
`null`.
   * Add the regression test 
`FastAlignedInnerCompactionWithDeletedValuePageTest`, which deletes a 
page-aligned time range on a subset of the value columns (1000 points per page) 
for both the single batch and the column-batched fast inner compaction.
   
   ## Tests
   
   * The new test fails on the old code with `expected:<null> but was:<1>` (the 
deleted value of `s1` at time 1 is revived) and passes with the fix.
   * `FastAlignedCrossCompactionTest`, 
`BatchedAlignedSeriesFastInnerCompactionTest`, 
`BatchedAlignedSeriesCrossSpaceCompactionTest`, 
`BatchedAlignedSeriesReadChunkCompactionTest`, 
`NewReadChunkCompactionPerformerWithAlignedSeriesTest`, 
`FastCompactionPerformerAlignedTest`, 
`FastCompactionPerformerWithEmptyPageTest`, `CompactionWithAllNullRowsTest`, 
`TableModelFastCompactionPerformerTest` and `FastNonAlignedCrossCompactionTest` 
pass.
   * `spotless:check` passes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to