yihua opened a new pull request, #20380: URL: https://github.com/apache/hudi/pull/20380
### Describe the issue this Pull Request addresses closes #20379 Meta sync detects dropped partitions by reading every completed clean on the active timeline, not just the ones since its last commit time synced. On a lagging table that puts the sync in a race with the archiver over cleans whose partition deletions earlier syncs already applied: when the archiver removes one mid-read, the sync fails for that cycle. ### Summary and Changelog - `TimelineUtils.getDroppedPartitions` now scans only the cleans since the last sync (requested after `lastCommitTimeSynced`, plus hollow cleans completed after `lastCommitCompletionTimeSynced`) when both watermarks are known, using the same filter as `getCommitsTimelineAfter`, extracted as `getInstantsSince`. A clean inside that window that is missing or unreadable fails the call, since the drop cannot be known without it; the best-effort skip of an archived clean stays only for the unbounded scan. - `HiveSyncTool`: when a timeline read fails during an incremental sync because the archiver removed instants the sync still needed (a `FileNotFoundException` anywhere in the cause chain), the same sync falls back to syncing all partitions from the storage listing instead of failing. - Tests: `TestTimelineUtils` covers the bounded window on both timeline layouts (table version 6 and current), a corrupt or missing clean inside it, the unbounded scan when the completion watermark is unknown, the hollow-clean rule, and the unchanged no-watermark skip; `TestHiveSyncToolPartitionSyncFallback` covers the fallback and the rethrow of any other read failure. ### Impact No public API or config change. Incremental meta sync reads fewer clean instants per cycle. ### Risk Level low. The new tests fail against the previous code: the bounded-scan test with `Failed to get partitions cleaned at [00003__...__clean__COMPLETED]`, the fallback test with `HoodieIOException: Could not read commit details`. Known and unchanged by this PR: the hollow filter is strict, so on timeline layout V1 (completion time is the file modification time, one second granularity on S3) a clean completing in the same second as the watermark is not re-read, as already holds for hollow commits; across a v6 to v8 upgrade the completion-time source changes, so the first post-upgrade window may miss or re-read one clean. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
