Thanks, Dmitri. I understand the cache-correctness concern. The question I’d like to explore is whether changing a storage endpoint should be allowed to change the content returned for the same table metadata file path. My view is that the content should remain unchanged. A table metadata should not be changed by how the FileIO client configured.
I think clarifying that expectation would help us decide whether the endpoint needs to be part of the cache key. Yufei On Tue, Oct 6, 2026 at 11:38 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Yufei, > > Thanks for starting this thread. > > I'd like to correct the problem statement, though. > > The question is not about the data consistency in different systems > (different endpoints), but about cache correctness. > > Since the storage endpoint identifies a service, Polaris can reasonably > expect that service to produce consistent data for the same object URI. > Subsequently, Iceberg data and metadata files are supposed to be immutable > by design. Therefore, Polaris can assume that the same Iceberg table file > loaded through the same endpoint and same URI produces the same data > every time. Therefore, it can be cached. > > If the endpoint is different, if may still resolve to the same storage > system, or it may not. There is some uncertainty, which, IMHO, makes > caching metadata invalid in that situation. More specifically, a cached > response from one endpoint cannot be used to substitute a response from > another endpoint even if the location URI (e.g. s3://something) is the > same. > > Cheers, > Dmitri. > > > > On Tue, Oct 6, 2026 at 1:42 PM Yufei Gu <[email protected]> wrote: > > > Hi folks, > > > > During review of PR #5328 <https://github.com/apache/polaris/pull/5328>, > > we > > encountered a question: is it valid for a storage endpoint change to make > > an existing table’s metadata URI return different content? > > > > For example, two S3-compatible stores contain > > s3://warehouse/orders/metadata/xxx.metadata.json with the same table > UUID: > > > > - Endpoint A’s metadata points to table metadata A. > > - Endpoint B’s metadata points to table metadata B. > > > > Changing the catalog’s endpoint from A to B changes the metadata returned > > without updating the table’s metadata pointer. Neither underlying object > > was modified. > > > > Does this violate Iceberg’s metadata immutability model, or is > immutability > > scoped to each underlying storage object? Are there legitimate migration > or > > recovery workflows that require this behavior? > > > > My understanding is that an existing metadata location should continue to > > identify the same content after storage configuration changes. Clarifying > > this contract would help determine whether metadata caching needs to > > account for storage endpoints. > > Yufei > > >
