RussellSpitzer commented on issue #11648: URL: https://github.com/apache/iceberg/issues/11648#issuecomment-5456028754
For position deletes we had two layouts: partition-scoped and file-scoped (write.delete.granularity). File-scoped files only contain deletes for a single data file, so caching the whole file never helps a second data file. Our default for Spark is FILE scoped. Partition-scoped files can contain deletes for many data files, but in practice they usually don’t grow that large. A write only sees the files modified in that job, and people rarely ran position-delete compaction (it landed well after v2 deletes). I don’t think I ever saw a position delete file get close to the 64 MB target. Equality deletes are the opposite: one file typically applies to every data file in the partition. The other issue is executor locality. The cache is per JVM, so you only win if two tasks that share a position delete file land on the same executor. For equality deletes that’s almost guaranteed because metric based exclusion is gonna be rare unless you keep your files sorted on the equality delete columns. For position deletes on a normal scan it’s mostly luck (unless we did more with scheduling / locality hints). Rewrite is a bit better within a file group, since those data files are read in one task, but groups in the same partition still go to different executors, so you still reload the delete file on each of them. My big suggestion here is we just add a parameter (`cache-delete-files`) so that users like yourself can turn that caching back on manually. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
