raunaqmorarka commented on PR #18359: URL: https://github.com/apache/iceberg/pull/18359#issuecomment-6219140000
> While the intention is good, I'm not sure how much effort would we like to put into optimising V2 deletes. > Especially, as this PR needs changes in BaseFile. v3 tables upgraded from v2 still read their existing position deletes until those are rewritten into DVs, so this path matters beyond v2 tables. The cost shows up as out-of-memory errors during planning. In the case behind this PR, one scan's index held about 10 GB for 4.3M deletes. This change cuts that by 60 to 70 percent. The `BaseFile` change is one package-private constructor overload, with no public API or format change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
