electrum commented on pull request #1499: URL: https://github.com/apache/iceberg/pull/1499#issuecomment-702997838
@rdblue Thanks for the detailed and thoughtful response. I'm embarrassed that I misread the specification and thought equality deletes were at the file level. At the partition level makes sense given how manifest files work. However, for their usage by a relational `DELETE` query, storing them in separate files is still very expensive at read time, since we have to do a file read for a single value. I can think of a couple of ways to address this cost: * Allow storing the equality delete inline in the manifest (we could have a strict or recommended size limit) * Share the same equality delete file across multiple partitions (is this allowed? how does cleanup work?) We might also allow predicates in the inline version, as the inline version would be limited to a small number of predicates. Since they are at the partition level, they are naturally filtered to partitions that existed at the time of the delete. Maybe think of this as a third type of delete operation, rather than as a special variant of equality delete. Equality deletes in external files for CDC operations by row ID is clever and makes sense, as it makes those operations far less expensive (no need to read the original files and join to find the matching rows). I had not considered that type of operation. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
