Hi Ashok, +1 on the proposal. Thanks for identifying the issue and providing a fix!
We don't need to perform a full validation when "write.metadata.path" is changed. Leaving historical metadata files in their original locations ensures immutability for current and historical snapshots. If we want to change historical file locations, a separate migration process would be required, which is beyond the scope of a single table commit. Yufei On Mon, Oct 5, 2026 at 5:25 AM Alex Dutra <[email protected]> wrote: > Hi Ashok, > > Your proposal sounds reasonable to me, +1. > > Thanks, > Alex > > On Fri, Oct 2, 2026 at 6:28 PM Ashok Borra via dev > <[email protected]> wrote: > > > > Hi All, > > > > Following the review discussion on #5582 > > <https://github.com/apache/polaris/pull/5582>, I'd like to bring this to > > the list. > > > > *Problem* > > > > Since #5115 <https://github.com/apache/polaris/pull/5115>, every table > > commit validates the locations of all direct metadata references in the > > table: the manifest list for every snapshot, plus the statistics and > > partition statistics files. Before that change, a regular append did not > > validate these locations. > > > > As a result, commit work grows with the table's snapshot history, even > when > > a commit adds just one snapshot. Tables with thousands of snapshots can > > trigger thousands of repeated path checks per commit. The full set of > paths > > is also included in each DEBUG message and in rejection errors. Together, > > this can lead to slow commits, disk pressure from log volume, and OOMs. > > > > *Proposal* > > > > References already accepted under the same storage restrictions do not > need > > to be checked again on every commit. I propose: > > > > 1. On a regular commit, validate only direct metadata references newly > > introduced relative to the base metadata: manifest lists for new > snapshots, > > new statistics and partition statistics files, and the new metadata file > > (as today). > > 2. Keep full validation for *create* and *register* operations, and for > > commits that change the table location - *write.metadata.path*. > > 3. Keep the parent-directory deduplication from #5582 > > <https://github.com/apache/polaris/pull/5582> for full-validation paths, > > with provider-aware parent path handling in StorageLocation, as suggested > > in the review. > > > > Thanks, > > Ashok >
