Priyadarshini-Mitra commented on PR #16967: URL: https://github.com/apache/iceberg/pull/16967#issuecomment-4961857970
> > I have pushed an update: renamed to ValidateTableIntegrity and rewrote the description to lead with integrity check. > > @Priyadarshini-Mitra yes so i think that's in line with the RepairTable that we were planning to work on, i dont see the need for another check? > > The only thing I'm not sure about Source-vs-destination mode. I feel it seems too low level for me. > > I know we have this mapping for one way to copy a table, but typical database system dont verify a copy of a table is same as another table by comparing file paths match with a prefix map. If you do anything on the table like expire snapshot or any compaction, this becomes invalid. Hence, I imagine table-copy verification depend on higher level things like row count, stats, etc. > > If we truly need a file by file comparison, I feel its just a basic spark job that selects from all_files + all_manifests + snapshots and check if those file exist (with prefix adjustment) on the target? Hence dont really see this being that needed in Iceberg itself, wdyt? Thanks @szehon-ho , good to hear the self-audit direction aligns with RepairTable. Happy to coordinate on that design. On source-vs-destination, a few things worth surfacing: Snapshot-scoped check : The action runs against a specific source metadata version and snapshot id, so the validation is anchored to a fixed source state rather than the source's current state. Compaction creates new snapshots on the source without removing the anchor; only expire_snapshots could invalidate the check, and only if the anchor snapshot is expired between copy and validation. In practice, validation runs immediately after copy, before any expiration. For DR verification : pin the source's metadata version and snapshot id when the replica is created, and validation confirms the destination still resolves against that snapshot's file set regardless of how the source has evolved since. File existence vs semantic equivalence : Row count and stats comparison verify data equivalence. File existence verifies that the destination's metadata is internally consistent (its manifests point at files that exist). Different failure modes: - Row count won't detect a destination whose metadata references files that don't exist. - File existence won't detect subtle content mismatches. Both are useful, this action covers the file-level integrity check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
