szehon-ho commented on PR #16967: URL: https://github.com/apache/iceberg/pull/16967#issuecomment-4961597519
> I have pushed an update: renamed to ValidateTableIntegrity and rewrote the description to lead with integrity check. @Priyadarshini-Mitra yes so i think that's in line with the RepairTable that we were planning to work on, i dont see the need for another check? The only thing I'm not sure about Source-vs-destination mode. I feel it seems too low level for me. I know we have this mapping for one way to copy a table, but typical database system dont verify a copy of a table is same as another table by comparing file paths match with a prefix map. It's too low-level and anything on the table like expire snapshot or compaction, this becomes invalid. Hence, usually these table verification depend on higher level things like row count. If we truly need a file by file comparison, I feel its just a basic spark job that selects from all_files + all_manifests + snapshots and check if those file exist (with prefix adjustment) on the target? Hence dont really see this being that needed in Iceberg itself, wdyt? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
