tom-s-powell opened a new pull request, #18011: URL: https://github.com/apache/iceberg/pull/18011
## Summary Avoid eagerly caching and counting the orphan-file dataset when `PrefixMismatchMode` is `IGNORE` or `DELETE`. The default `ERROR` behaviour remains unchanged. ## Why `findOrphanFiles` previously cached and counted the complete joined dataset for every prefix mismatch mode. This eager Spark job is required for `ERROR`, where all scheme and authority conflicts must be collected before any files are deleted. `IGNORE` and `DELETE` do not inspect the conflict accumulator. Eagerly materializing the dataset adds unnecessary work and cache pressure, which is particularly expensive for tables with very large file listings. ## How it works For `IGNORE` and `DELETE`, `findOrphanFiles` now returns the lazy dataset immediately after constructing the join and partition mapping. For `ERROR`, it retains the existing cache and count operation so conflicts are detected before deletion begins. Regression tests use a Spark accumulator to verify that `IGNORE` and `DELETE` do not evaluate the input dataset until the result is consumed. The change is applied to Spark 3.5, 4.0, 4.1, and 4.2. ## Testing - Ran the lazy evaluation and prefix conflict tests on all four Spark versions: 18/18 passed per version. - Ran the complete Spark 4.0 orphan-file action test class: 175 passed and 23 skipped. - Ran `spotlessCheck` on all affected modules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
