yihua opened a new issue, #20379: URL: https://github.com/apache/hudi/issues/20379
**Describe the problem you faced** Meta sync (`HiveSyncTool`, and Glue sync through it) computes dropped partitions with `TimelineUtils.getDroppedPartitions`, which reads every completed clean on the active timeline regardless of the last commit time synced. On a table whose sync lags, that read races the archiver over old cleans. When the archiver deletes one between the listing and the read, the sync fails for that cycle with `Failed to get partitions cleaned at ...` (on S3 the open is lazy, so the 404 surfaces as `IOException: Not an Avro data file` wrapping the `FileNotFoundException`, which the current direct-cause check does not recognize). Those old cleans carry nothing the sync still needs: their partition deletions were already applied by earlier syncs. The replace-commit and write scans in the same method are already bounded to instants after the last commit time synced; only the clean scan is not. **Expected behavior** The clean scan reads only the cleans since the last sync, a clean inside that window that cannot be read fails the sync loudly (the drop cannot be known without it), and if the archiver removes instants the sync still needs while it is running, that sync falls back to a full partition listing instead of failing. **Environment Description** - Hudi version: 1.x (master), also present on the 1.1 and 1.3 lines - Storage: S3, but any storage can hit the listing-then-read race -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
