Hi Yufei, > [...] the content should remain unchanged [...]
That may be so. However, Polaris does not control the data that exists in storage systems. If there is a real possibility that the data may be different, Polaris should not assume it is the same. At least not for caching purposes. Iceberg files stored in one particular system may change too. However, in that case the onus is on the user to prevent such changes to ensure logical data consistency. When the storage system is different, though, the guarantees about data file contents are much weaker. It's like having two files under the same name in two different disk drives. Cheers, Dmitri. On Tue, Oct 6, 2026 at 9:01 PM Yufei Gu <[email protected]> wrote: > Thanks, Dmitri. I understand the cache-correctness concern. > > The question I’d like to explore is whether changing a storage endpoint > should be allowed to change the content returned for the same table > metadata file path. My view is that the content should remain unchanged. A > table metadata should not be changed by how the FileIO client configured. > > I think clarifying that expectation would help us decide whether the > endpoint needs to be part of the cache key. > > Yufei > > > On Tue, Oct 6, 2026 at 11:38 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Yufei, > > > > Thanks for starting this thread. > > > > I'd like to correct the problem statement, though. > > > > The question is not about the data consistency in different systems > > (different endpoints), but about cache correctness. > > > > Since the storage endpoint identifies a service, Polaris can reasonably > > expect that service to produce consistent data for the same object URI. > > Subsequently, Iceberg data and metadata files are supposed to be > immutable > > by design. Therefore, Polaris can assume that the same Iceberg table file > > loaded through the same endpoint and same URI produces the same data > > every time. Therefore, it can be cached. > > > > If the endpoint is different, if may still resolve to the same storage > > system, or it may not. There is some uncertainty, which, IMHO, makes > > caching metadata invalid in that situation. More specifically, a cached > > response from one endpoint cannot be used to substitute a response from > > another endpoint even if the location URI (e.g. s3://something) is the > > same. > > > > Cheers, > > Dmitri. > > > > > > > > On Tue, Oct 6, 2026 at 1:42 PM Yufei Gu <[email protected]> wrote: > > > > > Hi folks, > > > > > > During review of PR #5328 <https://github.com/apache/polaris/pull/5328 > >, > > > we > > > encountered a question: is it valid for a storage endpoint change to > make > > > an existing table’s metadata URI return different content? > > > > > > For example, two S3-compatible stores contain > > > s3://warehouse/orders/metadata/xxx.metadata.json with the same table > > UUID: > > > > > > - Endpoint A’s metadata points to table metadata A. > > > - Endpoint B’s metadata points to table metadata B. > > > > > > Changing the catalog’s endpoint from A to B changes the metadata > returned > > > without updating the table’s metadata pointer. Neither underlying > object > > > was modified. > > > > > > Does this violate Iceberg’s metadata immutability model, or is > > immutability > > > scoped to each underlying storage object? Are there legitimate > migration > > or > > > recovery workflows that require this behavior? > > > > > > My understanding is that an existing metadata location should continue > to > > > identify the same content after storage configuration changes. > Clarifying > > > this contract would help determine whether metadata caching needs to > > > account for storage endpoints. > > > Yufei > > > > > >
