grantatspothero commented on PR #16939: URL: https://github.com/apache/iceberg/pull/16939#issuecomment-4791197871
Thanks for confirming. Yes the options to fix the behavior are: 1. Commit time validation: Fail fast during commit if an invalid delete file partition is detected. This is the best solution, but as you mentioned is expensive to do at commit time since it requires reading every manifest in the worst case. 2. Scan time filtering: If a referenced data file partition does not match the delete file partition, then do not associate the delete with the data file during the scan. This is less ideal because invalid commits are still allowed, but it enforces the spec is followed. It should be relatively cheap to implement because at scan time both the delete and data file partition value exist in memory. While (1) comes with tradeoffs, (2) seems like a pure improvement over the existing state since it adds spec-correctness with no additional cost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
