Neuw84 commented on PR #18258: URL: https://github.com/apache/iceberg/pull/18258#issuecomment-6083223578
Thanks @kinolaev, both points are fair. Pushed 30d281b: - **Opt-in:** merged-set caching is now behind `BaseDeleteLoader.cacheEqualityDeleteSets()` (default `false`); Spark 4.2 enables it with `spark.sql.iceberg.executor-cache.equality-delete-sets.enabled` (default `false`). Without it the loader behaves exactly as on `main`, one cache entry per delete file, so the partially-overlapping case you describe doesn't regress. The docs entry calls out that trade-off. - **Key length:** the key is now a SHA-256 digest of the projection and the sorted locations, so it's constant-size regardless of how many delete files a task references. I agree #16956's snapshot-wide `EqualityDeletes` structure is the better long-term shape for overlapping sets; this keeps a small, opt-in win for the identical-sets case in the meantime, and the two could combine as you suggest. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
