yihua opened a new pull request, #20380: URL: https://github.com/apache/hudi/pull/20380
### Describe the issue this Pull Request addresses closes #20379 Meta sync detects dropped partitions by reading every completed clean on the active timeline, not just the ones since its last commit time synced. On a lagging table that puts the sync in a race with the archiver over cleans whose partition deletions earlier syncs already applied: when the archiver removes one mid-read, the sync fails for that cycle. ### Summary and Changelog - `TimelineUtils.getDroppedPartitions` now scans only the cleans since the last sync (requested after `lastCommitTimeSynced`, plus hollow cleans completed after `lastCommitCompletionTimeSynced`) when both watermarks are known, using the same filter as `getCommitsTimelineAfter`, extracted as `getInstantsSince`. A clean inside that window that is missing or unreadable fails the call, since the drop cannot be known without it; the best-effort skip of an archived clean stays only for the unbounded scan. - `HiveSyncTool`: when a timeline read fails during an incremental sync because the archiver removed instants the sync still needed (a `FileNotFoundException` in the cause chain, or the last commit time synced no longer on a freshly listed active timeline), the same sync falls back to syncing all partitions from the storage listing instead of failing. The fresh listing does not replace the timeline the sync works from, so the last commit time synced it records still matches what it synced. - Tests: `TestTimelineUtils` covers the bounded window, a corrupt or missing clean inside it, the unbounded scan when the completion watermark is unknown, the hollow-clean rule, and the unchanged no-watermark skip; `TestHiveSyncToolPartitionSyncFallback` covers both fallback triggers and the rethrow when the watermark is still active. ### Impact No public API or config change. Incremental meta sync reads fewer clean instants per cycle. ### Risk Level low. The new tests fail against the previous code: the bounded-scan test with `Failed to get partitions cleaned at [00003__...__clean__COMPLETED]`, the fallback test with `HoodieIOException: Could not read commit details`. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
