dramaticlly commented on PR #18512: URL: https://github.com/apache/iceberg/pull/18512#issuecomment-6197593399
@amogh-jahagirdar do you think this make sense? Today in iceberg-core, I saw there's difference on how we handle the runtime problem in snapshot expiration (some metadata tracked by iceberg does not present on the file system), in [IncrementalFileCleanup](https://appleinc.webex.com/appleinc/j.php?MTID=m6febe6921fd010a4db2a7fe0911aa5da) we apply the best effort while in [ReachableFileCleanup](https://github.com/apache/iceberg/blob/39e35d1a87c0a6a9329d8ba74a45d587808598fa/core/src/main/java/org/apache/iceberg/ReachableFileCleanup.java#L118) we will throw if manifest-list is not found. There's also an attempt in #18447 tried to fix in the spark action. I hope by adding ignoreMissingFiles to ExpireSnapshots interface, we can have a better way do standardize across the common problem, especially for large tables where orphan removal and fs listing is challenging at scale. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
