Hi All, Following the review discussion on #5582 <https://github.com/apache/polaris/pull/5582>, I'd like to bring this to the list.
*Problem* Since #5115 <https://github.com/apache/polaris/pull/5115>, every table commit validates the locations of all direct metadata references in the table: the manifest list for every snapshot, plus the statistics and partition statistics files. Before that change, a regular append did not validate these locations. As a result, commit work grows with the table's snapshot history, even when a commit adds just one snapshot. Tables with thousands of snapshots can trigger thousands of repeated path checks per commit. The full set of paths is also included in each DEBUG message and in rejection errors. Together, this can lead to slow commits, disk pressure from log volume, and OOMs. *Proposal* References already accepted under the same storage restrictions do not need to be checked again on every commit. I propose: 1. On a regular commit, validate only direct metadata references newly introduced relative to the base metadata: manifest lists for new snapshots, new statistics and partition statistics files, and the new metadata file (as today). 2. Keep full validation for *create* and *register* operations, and for commits that change the table location - *write.metadata.path*. 3. Keep the parent-directory deduplication from #5582 <https://github.com/apache/polaris/pull/5582> for full-validation paths, with provider-aware parent path handling in StorageLocation, as suggested in the review. Thanks, Ashok
