Priyadarshini-Mitra opened a new issue, #16868: URL: https://github.com/apache/iceberg/issues/16868
## Proposal Add a `ValidateRewriteTablePath` Spark action that verifies every metadata, data, and delete file the source table references is present at the destination. Operates against any source/destination pair where the destination is supposed to be a copy of the source — most commonly the output of `RewriteTablePath`, but also DR replicas, migration targets, backups, or any out-of-band copy. The validator treats the source table as the source of truth: it walks source metadata to enumerate expected files, applies any configured prefix rewrite, and verifies each path at the destination. It does not rely on destination metadata to enumerate expected files, so a destination with internally inconsistent metadata still gets a correct missing-file report. Tracked across three sequential PRs: - [ ] API contract (interface + factory + Result base) - [ ] Spark implementation + core helpers + exception - [ ] `validate_rewrite_table_path` Spark procedure + user docs ## Two common uses **Verify a copy before registering it** — confirm the destination contains every expected file before registering the new metadata.json with a catalog. A failed validation prevents an incomplete copy from becoming a live table. **Audit any source/destination pair for sync** — confirm a destination still references every file the source does. Catches missing or stale copies before they cause data loss on failover, wrong query results from a migrated table, or failed restores. ## Out of scope - Content equivalence — validates file existence, not file content. - Destination metadata internal consistency — separate concern. ## Risk - Low risk — additive, opt-in feature with no behavioral change for existing tables or actions. - New action; does not modify any existing code paths. - Read-only at the destination — only performs file existence checks, never writes. ### Query engine Spark ### Willingness to contribute - [X] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
