dramaticlly commented on PR #18546: URL: https://github.com/apache/iceberg/pull/18546#issuecomment-6321677393
thanks @rdblue for the well thought explanation, current validation logic always takes me some time to digest and I am excited about the new cached validator. With colocated data and deletion vector colocate in the V4 tracked file, it does simplify some of the validations. With respect to the how delete within the overwriteFiles are committed, I do agree that delete by reference and delete by row filters are intended for different purposes. Sharing one data point I worked on in the past which rely on row filter to drop a whole region without listing the live entries first as it's evaluated directly in manifest filter manager. This typically happen outside the regular engine writes where scan-then-commit is not readily available. I can buy the argument for making `overwriteByRowFilter` and `conflictDetectionFilter` mutually exclusive, since in the long run delete by file reference can express the same thing. But I would callout the behavior changes (maybe in dev list) and provide recommendations. @stevenzwu curious if you see the need to keep the 2nd validator ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
