ai-yang opened a new issue, #10755: URL: https://github.com/apache/rocketmq/issues/10755
## RocketMQ version `develop` at `a06836dd564e5e43115493f775626cf98d51d10e` ## Describe the bug `ConsumeQueue.truncateDirtyLogicFiles` updates the ConsumeQueue positions but skips truncating the associated `ConsumeQueueExt` during every normal partial-truncation path. The method tracks the last retained extension address in `maxExtAddr`, and its tail calls: ```java consumeQueueExt.truncateByMaxAddress(maxExtAddr); ``` However, scanning returns directly when it reaches the first dirty CQ entry, the end of a full valid CQ file, or an unwritten slot. Those returns bypass the extension truncation. The tail is only reached after all CQ files have been deleted or no CQ file exists; in that case `maxExtAddr` remains `1`, which is not an extension address, so `truncateByMaxAddress` is a no-op. ## Deterministic reproduction 1. Create a file-backed `ConsumeQueue` with `enableConsumeQueueExt=true`. 2. Write four CQ entries with physical offsets `0`, `100`, `200`, and `300`, using distinct tag codes so that four CQExt units are created. 3. Record the raw extension addresses stored by CQ entries 2 and 3. 4. Call `truncateDirtyLogicFiles(200)`, retaining entries 0 and 1. 5. Write replacement entries at CQ offsets 2 and 3 with different tag codes. 6. Flush, load a second `ConsumeQueue` from the same directory, and call `recover()`. 7. Read the extension unit at the original address of entry 2. Expected: - CQExt is truncated together with CQ. - Replacement entries reuse the truncated extension addresses. - After reload, the original address of entry 2 contains the replacement tag code. Actual: - The old extension units remain readable after CQ truncation. - Replacement extension units are appended after the orphaned units. - Recovery retains the orphaned units because the newly appended CQ entries now reference later extension addresses. The controlled regression test is named: ```text ConsumeQueueTest#testTruncateDirtyLogicFilesTruncatesConsumeQueueExtAndSurvivesReload ``` It uses no sleeps, randomized scheduling, or external services. On the latest unmodified `develop` baseline under JDK 8, it failed at the same business assertion in 5/5 isolated Maven/JUnit processes: expected replacement tag code `202`, but the original orphan still returned `102`. ## Impact After CommitLog/CQ rollback followed by rebuilding, unreachable ConsumeQueueExt units remain between retained and replacement units. This leaks persistent store space and cannot be repaired by a later restart, because recovery truncates only after the newest referenced extension address. ## Suggested fix - Route every normal scan exit through the CQExt truncation tail instead of returning early. - Continue deleting wholly dirty tail CQ files until the last retained entry is found. - Treat an empty first slot as an empty tail file and continue scanning the preceding retained file. - If a retained CQ entry references CQExt, truncate after its last retained extension address; if no retained CQ entry references CQExt, physically clear the extension files so stale bytes cannot reappear during reload/recovery. - Make the full-clear path recoverable when an extension mapped file is still referenced: retain failed mappings, block address reuse while cleanup is pending, and allow a later retry after the reader releases the buffer. - When recovery finds no CQ file, clear any loaded CQExt files so a crash between CQ deletion and extension cleanup self-heals on restart. - Preserve the non-destructive `deleteFile=false` overload by not physically truncating CQExt while detached CQ files remain on disk. - Add partial-file, empty/full-tail-file, full-truncation, cleanup-failure/retry, replacement-write, and reload/recovery regression coverage. ## Related work checked - #2570/#2572 only change when recovery invokes `truncateDirtyLogicFiles`; they do not truncate CQExt. - #2571/#2573 discuss the physical-offset boundary condition, not extension cleanup. - #10417/#10418 fixes stale minimum offsets after all CQ files are deleted; it does not reach or repair CQExt tail truncation. - Open #9728/#9729 handles an empty last mapped file in `correctMinOffset`, a different method and postcondition. - #6343 concerns deleting the CQExt directory when a topic is deleted. - #10733 touches a different method in `ConsumeQueue` (`estimateMessageCount`) and does not change truncation. Immediate pre-publication searches across open and closed issues and open, closed, and merged pull requests for `ConsumeQueueExt`, `truncateDirtyLogicFiles`, `truncateByMaxAddress`, extension truncation, and orphaned extension data found no equivalent report, claimed fix, or assignee. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
