rdblue commented on pull request #1499: URL: https://github.com/apache/iceberg/pull/1499#issuecomment-703030934
> However, for their usage by a relational DELETE query, storing them in separate files is still very expensive at read time, since we have to do a file read for a single value. For a relational DELETE query, I would either encode it as a global equality delete, or scan through data files and use position deletes. Position deletes are the best for readers, so we generally want `DELETE FROM` to use them. Equality deletes are mostly for cases where you can't find where the matching rows are, like streaming. And just to be clear, equality delete files can have an arbitrary number of deletes, one per row. While in the worst case it could be a file read for a single delete, we would not expect that in normal operation. I don't think I would add deletes inline in manifests. That would require a lot of complication in manifests and would greatly expand table-level metadata in some use cases. I'd prefer to keep deletes distributed like data files, instead of centralized like table metadata. The idea to share some equality delete files across partitions has occurred to me, but I think it is reasonable to either use a global delete (that is written once and shared) or a partition-level delete. We can also write a global delete once and have a background process rewrite to partition-level deletes. If you need to write fast, then use a global. For more targeted reads, use partition-level. And better yet, use position deletes to minimize overhead. I really don't see equality deletes being used for SQL cases, because position deletes are so much better for readers. If there isn't time to find the data, use a global delete, but then rewrite to position deletes -- not equality -- because that's better as long as you're doing work in the background. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
