Hi all, I think many valid concerns have already been mentioned on this thread.
A fair summary of the assumptions could be: 1. Metadata files remain immutable within the storage context. 2. Polaris and clients resolve their locations into the same logical storage namespace. Neither assumption follows from the metadata URI and Polaris storage configuration remaining unchanged. Using the AWS SDK as an example: object storage is accessed from both Polaris and its clients. The SDK supports endpoint configuration through environment variables, Java system properties and configuration files. An explicit endpoint override, like Iceberg's `s3.endpoint`, takes precedence. But where no effective explicit endpoint is supplied, Polaris and clients can pick up different endpoints from their respective environments despite using the same catalog configuration and object URI. Different endpoints may legitimately access the same logical storage namespace. For example, clients might use nearby replica endpoints, provided those replicas preserve the consistency required by the workload. But identical object URIs do not establish that equivalence. Similarly, Iceberg assumes metadata and other files are immutable once written. That is an operating contract; it does not itself prevent an actor with sufficient storage permissions from replacing their contents. The above doesn't make configuration-aware caching pointless. Caching relies on the immutability contract within a stable storage context, and changes to Polaris storage configuration should prevent reuse across contexts whose equivalence Polaris cannot establish. That is a useful safeguard, but it cannot detect every change in the underlying storage or its resolution. Even perfect cache invalidation following Polaris configuration changes cannot establish the second assumption. Disabling caching would not solve it either: Polaris could freshly read metadata from A while the client accesses B. There's one more thing to the story: It's not just about the endpoint. Metadata caching may also need to account for the FileIO implementation, region and internal endpoint, while vended-credential caching needs to distinguish the role ARN, STS endpoint, external ID and requested write grants. The need to coordinate catalog state with independently accessed storage is inherent in Iceberg’s architecture, and Iceberg REST carries that architecture forward. Configuration-aware caching is a useful safeguard within that model, but cannot establish end-to-end storage consistency on its own. Robert On Wed, Oct 7, 2026 at 7:31 AM Jean-Baptiste Onofré <[email protected]> wrote: > Hi guys, > > Both points make sense, but they address different levels imho: > - To Yufei's point, Iceberg assumes metadata files are immutable once > written (we could argue about that :)). Ideally, the metadata URI acts as a > content identifier, and the storage endpoint is just the "access route". > - To Dimitri's point, Polaris cannot enforce what external object stores > actually contain. A s3:// URI is not a content-addressable hash (like a git > commit SHA). Pointing to a different endpoint (during failover, migration > between stores, different environments, ...) could resolve to > different data or a replica. Serving cached metadata from Endpoint A after > an operator pointed the catalog to Endpoint B breaks "freshness" guarantee. > > Also, credential vending is tied to the current storage configuration. If > Polaris returned cached metadata from Endpoint A while vending credentials > for Endpoint B, clients would execute commits and read data against > Endpoint B based on metadata from Endpoint A, with the risk on > inconsistencies. > > Since storage endpoint or configuration updates are rare control-plane > operations, scoping the cache to the storage configuration / entity version > costs almost nothing in cache hit rate, while guaranteeing correctness. > > So, while I agree with Yufei on the theorical immutability principle with a > given storage boudary, I agree with Dmitri that Polaris must be defensive > and scope cache entries to the storage configuration/version. > > Regards > JB > > On Wed, Oct 7, 2026 at 3:15 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Yufei, > > > > > [...] the content should remain unchanged [...] > > > > That may be so. However, Polaris does not control the data that exists in > > storage systems. > > > > If there is a real possibility that the data may be different, Polaris > > should not assume it is the same. At least not for caching purposes. > > > > Iceberg files stored in one particular system may change too. However, in > > that case the onus is on the user to prevent such changes to ensure > logical > > data consistency. > > > > When the storage system is different, though, the guarantees about data > > file contents are much weaker. It's like having two files under the same > > name in two different disk drives. > > > > Cheers, > > Dmitri. > > > > On Tue, Oct 6, 2026 at 9:01 PM Yufei Gu <[email protected]> wrote: > > > > > Thanks, Dmitri. I understand the cache-correctness concern. > > > > > > The question I’d like to explore is whether changing a storage endpoint > > > should be allowed to change the content returned for the same table > > > metadata file path. My view is that the content should remain > unchanged. > > A > > > table metadata should not be changed by how the FileIO client > configured. > > > > > > I think clarifying that expectation would help us decide whether the > > > endpoint needs to be part of the cache key. > > > > > > Yufei > > > > > > > > > On Tue, Oct 6, 2026 at 11:38 AM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi Yufei, > > > > > > > > Thanks for starting this thread. > > > > > > > > I'd like to correct the problem statement, though. > > > > > > > > The question is not about the data consistency in different systems > > > > (different endpoints), but about cache correctness. > > > > > > > > Since the storage endpoint identifies a service, Polaris can > reasonably > > > > expect that service to produce consistent data for the same object > URI. > > > > Subsequently, Iceberg data and metadata files are supposed to be > > > immutable > > > > by design. Therefore, Polaris can assume that the same Iceberg table > > file > > > > loaded through the same endpoint and same URI produces the same data > > > > every time. Therefore, it can be cached. > > > > > > > > If the endpoint is different, if may still resolve to the same > storage > > > > system, or it may not. There is some uncertainty, which, IMHO, makes > > > > caching metadata invalid in that situation. More specifically, a > cached > > > > response from one endpoint cannot be used to substitute a response > from > > > > another endpoint even if the location URI (e.g. s3://something) is > the > > > > same. > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > > > > > > > > > On Tue, Oct 6, 2026 at 1:42 PM Yufei Gu <[email protected]> > wrote: > > > > > > > > > Hi folks, > > > > > > > > > > During review of PR #5328 < > > https://github.com/apache/polaris/pull/5328 > > > >, > > > > > we > > > > > encountered a question: is it valid for a storage endpoint change > to > > > make > > > > > an existing table’s metadata URI return different content? > > > > > > > > > > For example, two S3-compatible stores contain > > > > > s3://warehouse/orders/metadata/xxx.metadata.json with the same > table > > > > UUID: > > > > > > > > > > - Endpoint A’s metadata points to table metadata A. > > > > > - Endpoint B’s metadata points to table metadata B. > > > > > > > > > > Changing the catalog’s endpoint from A to B changes the metadata > > > returned > > > > > without updating the table’s metadata pointer. Neither underlying > > > object > > > > > was modified. > > > > > > > > > > Does this violate Iceberg’s metadata immutability model, or is > > > > immutability > > > > > scoped to each underlying storage object? Are there legitimate > > > migration > > > > or > > > > > recovery workflows that require this behavior? > > > > > > > > > > My understanding is that an existing metadata location should > > continue > > > to > > > > > identify the same content after storage configuration changes. > > > Clarifying > > > > > this contract would help determine whether metadata caching needs > to > > > > > account for storage endpoints. > > > > > Yufei > > > > > > > > > > > > > > >
