Hi guys, Both points make sense, but they address different levels imho: - To Yufei's point, Iceberg assumes metadata files are immutable once written (we could argue about that :)). Ideally, the metadata URI acts as a content identifier, and the storage endpoint is just the "access route". - To Dimitri's point, Polaris cannot enforce what external object stores actually contain. A s3:// URI is not a content-addressable hash (like a git commit SHA). Pointing to a different endpoint (during failover, migration between stores, different environments, ...) could resolve to different data or a replica. Serving cached metadata from Endpoint A after an operator pointed the catalog to Endpoint B breaks "freshness" guarantee.
Also, credential vending is tied to the current storage configuration. If Polaris returned cached metadata from Endpoint A while vending credentials for Endpoint B, clients would execute commits and read data against Endpoint B based on metadata from Endpoint A, with the risk on inconsistencies. Since storage endpoint or configuration updates are rare control-plane operations, scoping the cache to the storage configuration / entity version costs almost nothing in cache hit rate, while guaranteeing correctness. So, while I agree with Yufei on the theorical immutability principle with a given storage boudary, I agree with Dmitri that Polaris must be defensive and scope cache entries to the storage configuration/version. Regards JB On Wed, Oct 7, 2026 at 3:15 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Yufei, > > > [...] the content should remain unchanged [...] > > That may be so. However, Polaris does not control the data that exists in > storage systems. > > If there is a real possibility that the data may be different, Polaris > should not assume it is the same. At least not for caching purposes. > > Iceberg files stored in one particular system may change too. However, in > that case the onus is on the user to prevent such changes to ensure logical > data consistency. > > When the storage system is different, though, the guarantees about data > file contents are much weaker. It's like having two files under the same > name in two different disk drives. > > Cheers, > Dmitri. > > On Tue, Oct 6, 2026 at 9:01 PM Yufei Gu <[email protected]> wrote: > > > Thanks, Dmitri. I understand the cache-correctness concern. > > > > The question I’d like to explore is whether changing a storage endpoint > > should be allowed to change the content returned for the same table > > metadata file path. My view is that the content should remain unchanged. > A > > table metadata should not be changed by how the FileIO client configured. > > > > I think clarifying that expectation would help us decide whether the > > endpoint needs to be part of the cache key. > > > > Yufei > > > > > > On Tue, Oct 6, 2026 at 11:38 AM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi Yufei, > > > > > > Thanks for starting this thread. > > > > > > I'd like to correct the problem statement, though. > > > > > > The question is not about the data consistency in different systems > > > (different endpoints), but about cache correctness. > > > > > > Since the storage endpoint identifies a service, Polaris can reasonably > > > expect that service to produce consistent data for the same object URI. > > > Subsequently, Iceberg data and metadata files are supposed to be > > immutable > > > by design. Therefore, Polaris can assume that the same Iceberg table > file > > > loaded through the same endpoint and same URI produces the same data > > > every time. Therefore, it can be cached. > > > > > > If the endpoint is different, if may still resolve to the same storage > > > system, or it may not. There is some uncertainty, which, IMHO, makes > > > caching metadata invalid in that situation. More specifically, a cached > > > response from one endpoint cannot be used to substitute a response from > > > another endpoint even if the location URI (e.g. s3://something) is the > > > same. > > > > > > Cheers, > > > Dmitri. > > > > > > > > > > > > On Tue, Oct 6, 2026 at 1:42 PM Yufei Gu <[email protected]> wrote: > > > > > > > Hi folks, > > > > > > > > During review of PR #5328 < > https://github.com/apache/polaris/pull/5328 > > >, > > > > we > > > > encountered a question: is it valid for a storage endpoint change to > > make > > > > an existing table’s metadata URI return different content? > > > > > > > > For example, two S3-compatible stores contain > > > > s3://warehouse/orders/metadata/xxx.metadata.json with the same table > > > UUID: > > > > > > > > - Endpoint A’s metadata points to table metadata A. > > > > - Endpoint B’s metadata points to table metadata B. > > > > > > > > Changing the catalog’s endpoint from A to B changes the metadata > > returned > > > > without updating the table’s metadata pointer. Neither underlying > > object > > > > was modified. > > > > > > > > Does this violate Iceberg’s metadata immutability model, or is > > > immutability > > > > scoped to each underlying storage object? Are there legitimate > > migration > > > or > > > > recovery workflows that require this behavior? > > > > > > > > My understanding is that an existing metadata location should > continue > > to > > > > identify the same content after storage configuration changes. > > Clarifying > > > > this contract would help determine whether metadata caching needs to > > > > account for storage endpoints. > > > > Yufei > > > > > > > > > >
