Priyadarshini-Mitra commented on PR #16967: URL: https://github.com/apache/iceberg/pull/16967#issuecomment-4948313563
> @Priyadarshini-Mitra , taking a little more look at this, you use the source to get a list of corresponding files that you want to verify exist in the target, is that it? > > doesnt the RewriteTablePath action give you the file list (both source and destination), that the copy tool will then copy? i still feel the external copy tool should be the one to check that all the source files made it to the destination. > > Or put another way, this is trying to validate copy , which is outside Iceberg. Its not really validating 'rewrite table path' which is just the pre-req of copy. Thanks @szehon-ho for your review The action is an integrity check between two Iceberg tables: does the destination have every source-referenced file ? Pure Iceberg semantics. 1. Validating referential integrity requires Iceberg semantics (manifest chains, DVs, statistics files) 2. Source-driven validation against Iceberg's own metadata is the ground truth On the name I think "ValidateTablePath" would be clearer since this isn't RewriteTablePath-specific. Happy to rename. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
