yadavay-amzn commented on issue #17206: URL: https://github.com/apache/iceberg/issues/17206#issuecomment-4996466854
Thanks for taking a look, Russell. You're right about the read path. Breaking hard on a spec violation is the correct default, and having the reader silently assume the DVs are all valid and safe to merge is an assumption I don't think we can stand behind either. I'll drop the read-time merge from #17207 so the hard fail at index build stays as it is. I would like to get your thoughts on the following. Reverting to a pre-corrupt snapshot is the right first answer and covers most cases. The two spots where rollback alone seems to fall short are (a) the pre-corrupt snapshot may have already been expired, so there's nothing left to roll back to, and (b) rollback also discards any legitimate commits written after the corruption point, which can be a lot of data. For just those cases, would an explicit, opt-in repair action be something you'd be open to? The idea would be a maintenance op the user runs knowingly, not anything on the read path, that writes a new spec-compliant snapshot with a single DV per data file. Union would be the resolution, and its main safety property is that it can only over-delete rather than resurrect a row that was already deleted, so it's the safer direction if we have to pick one automatically. That keeps the invariant intact everywhere by default and moves the "I accept this merge" decision to a human running maintenance rather than the reader. If you think rollback plus manual repair already covers this well enough, I'm happy to close #17207 and leave the hard fail as the whole story. Mostly want to check whether the repair-action direction is worth pursuing in your mind. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
