Hi Adam, Could you give some more details, please, about where you think the Iceberg spec disallows multiple storage per table?
I never thought it was not allowed. Perhaps it was inconvenient or impractical in existing clients, but the spec itself seems to be fairly unbiased with respect to actual storage technology... Perhaps I missed something, though. Thanks, Dmitri. On Thu, Jun 18, 2026 at 7:42 AM Adam Szita <[email protected]> wrote: > Hi Yufei, > > Regarding #2: > Would opting for a simple (assuming per table) storageConfigName or > storageConfigId > support future extensibility? > Asking from a multiple-storage-per-table perspective as it's something that > the Iceberg spec does allow IMHO. > > Cheers, > Adam > > On Wed, 17 Jun 2026 at 00:37, Yufei Gu <[email protected]> wrote: > > > Hi folks, thanks for the discussion. They are really helpful! > > > > I think there are two separate questions here: > > > > 1. How should we store storage configurations? As catalog properties or > as > > separate entities? > > I'd lean toward separate entities in the long term. It provides a cleaner > > boundary between a catalog and its storage configurations, avoids > bloating > > the catalog object as the number of configs grows, and gives us a better > > foundation for future access control since entity level permissions are > > easier to reason about. > > > > 2. How should a table or namespace refer to a storage configuration? > > I'd prefer storing a simple reference on the entity itself rather than > > introducing a separate relationship table. Both approaches support > > hierarchical lookup and inheritance when a storage config is not > specified > > at the current level. However, I don't think we should allow multiple > > storage configs to be associated with a single table or namespace. Given > > that cardinality, a simple storageConfigName or storageConfigId field > seems > > sufficient and easier to understand than a dedicated relationship model. > > > > For example: > > > > Catalog > > ├─ entity: StorageConfigA > > ├─ entity: StorageConfigB > > └─ entity: StorageConfigC > > > > Table1 storage property -> StorageConfigA > > Table2 storage property -> StorageConfigB > > Table3 storage property -> StorageConfigC > > > > To me, this keeps the model simple while still supporting shared storage > > configurations and future extensibility. > > > > Yufei > > > > > > On Mon, Jun 15, 2026 at 7:05 AM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi Robert, > > > > > > The distinction between "attached" and "referenced" config makes sense. > > > > > > My impression from JB's message on Mar 19 in this thread [1] is that JB > > is > > > specifically proposing the "referenced" model, though. > > > > > > From my POV, both approaches can work, but the "referenced" model might > > be > > > more intuitive to users. > > > > > > [1] https://lists.apache.org/thread/40hxpc7xyhr9hv5x70srk6j5zq1odsy3 > > > > > > Cheers, > > > Dmitri. > > > > > > On Mon, Jun 15, 2026 at 7:18 AM Robert Stupp <[email protected]> wrote: > > > > > > > Hi all, > > > > > > > > AFAIK the current Polaris code appears to resolve the storage > > > > configuration from the leaf (table), upwards via the namespace > elements > > > to > > > > the catalog, choosing the first one found. > > > > I am not sure whether that functionality is already used anywhere? > > > > Being able to use a storage configuration on a namespace or a leaf > > > appears > > > > appealing. > > > > > > > > OTOH I see a demand for named storage configurations. > > > > > > > > I wonder whether the mechanism to derive the storage configuration > > could > > > > support both "directly attached" and "referenced" storage > > configurations: > > > > A namespace element or leaf can have either a "directly attached" > > storage > > > > configuration or a reference to a named one. > > > > > > > > We'd need management-ish APIs to handle the storage configurations > for > > a > > > > namespace or leaf. > > > > These APIs, which do not exist yet, could start with "directly > > attached" > > > > storage configurations and once the authz questions are resolved, > > extend > > > to > > > > include named, shared storage configurations. > > > > > > > > Thoughts? > > > > > > > > Robert > > > > > > > > > > > > On Thu, Jun 11, 2026 at 7:28 PM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > To recap the discussion from the Community Sync today: My > > understanding > > > > is > > > > > that the consensus was: > > > > > > > > > > * Defer AuthZ aspects for later discussion. > > > > > > > > > > * Allow multiple Storage Config objects per Catalog. If more than > one > > > > > Storage Config is present, they are to have different (within the > > > > catalog) > > > > > names. Rishi to open a new PR for this (scoped to REST API + Sever > > > > > changes). > > > > > > > > > > * Defer CLI changes until we have the full end-to-end system > working > > > (it > > > > > should be usable via curl) > > > > > > > > > > * PR 4023 is on hold for now. It is to be rebased after adding > > support > > > > for > > > > > multiple Storage Configs > > > > > > > > > > * The connection between storage names and storage access > credentials > > > is > > > > to > > > > > remain unchanged for now. It is effectively a deployment time > concern > > > > given > > > > > the current OSS code. > > > > > > > > > > Please correct / add if I missed anything. > > > > > > > > > > From my POV, I think we still need to discuss how exactly entities > > > > (tables, > > > > > views) will link to Storage Config before we can resume PR 4023. > > > > > > > > > > I propose storing the config name in the Entity properties (not in > > > Table > > > > > Metadata properties). > > > > > > > > > > If clients also need a way to define it while interacting with > > Polaris > > > > over > > > > > the IRC API, let's discuss how that can be done. > > > > > > > > > > WDYT? > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Mon, Jun 8, 2026 at 2:17 PM Dmitri Bourlatchkov < > [email protected] > > > > > > > > wrote: > > > > > > > > > > > Hi Srinivas, > > > > > > > > > > > > I see that PR 4023 is active again. I'll try and post a fresh > > review > > > > > soon. > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Thu, Mar 19, 2026 at 5:41 AM Srinivas Rishindra < > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > >> Hi JB and Dmitri, > > > > > >> > > > > > >> I raised PR #4023 <https://github.com/apache/polaris/pull/4023> > > > > > yesterday > > > > > >> as part of the broader per-table storage config effort, and I > > > believe > > > > it > > > > > >> implements exactly what you are proposing. > > > > > >> > > > > > >> Building on the named storage configs introduced in PR #3409 > > > > > >> <https://github.com/apache/polaris/pull/3409>, this PR allows > > > > > namespaces > > > > > >> and tables to specify a polaris.storage.name property. This > acts > > as > > > > the > > > > > >> "identifier" JB mentioned, allowing credential vending to > resolve > > > the > > > > > >> correct, named storage config at runtime without introducing new > > > > > >> management > > > > > >> APIs. > > > > > >> > > > > > >> Could you please take a look at the PR and let me know if this > > > aligns > > > > > with > > > > > >> what you are looking for? > > > > > >> > > > > > > > > > > > > > > > > > > > > >
