Thanks, Rishi! This looks good overall. I'm good with Option B. Catalog scoped storage configurations also make sense to me. I agree that having a dedicated entity gives us a cleaner model for authorization, versioning, and future extensions.
Regarding the binding issue, I wonder if we can keep things simpler by allowing a storage configuration to be specified during table creation(for example, specified in a table property, so that engines like Spark don't have to adopt a new Polaris-specific plugin), while continuing to inherit the parent's configuration by default. The pending binding approach seems more complicated, especially around renames, stale bindings, and cleanup. I'd prefer to avoid introducing that complexity unless we have a strong use case that requires it. I'm good with moving forward with the design doc based on Option B. Yufei On Sun, Oct 25, 2026 at 8:49 PM Srinivas Rishindra <[email protected]> wrote: > Hi Yufei, > > Thanks for writing this up. I went through your points and I agree: let's > skip Phase 1 and make storage configurations their own entities. > > On your points: > > 1. Default base location. This goes beyond create-table. With default > settings a namespace can't have a custom location, so every namespace > location is derived from the catalog's default-base-location. A > namespace > bound to an Azure configuration in an S3 catalog would get an s3:// > location, and every table in it would fail the allowed-locations check. > The > only workaround is turning on ALLOW_NAMESPACE_CUSTOM_LOCATION and > setting > each namespace's location by hand. An entity doesn't fix this by itself > either. I'd give each configuration an optional base location, use it > for > namespaces and tables bound before they're created (more below), and > refuse > to bind an existing namespace or table whose location isn't inside the > configuration's allowed locations. > > 2, 3 and 5. Agreed. On 3: #5721 already moved to set/remove fields, but > it's > still one catalog entity with one version, so two admins editing different > configurations conflict. > > 4. Access control. Agreed, and this is the real cost of skipping Phase > 1: we deferred AuthZ in June to unblock it, and now we need it up front. > I'd follow the policy model: create, read, write, drop and list > privileges > on the configuration, and attach as a pair (STORAGE_CONFIG_ATTACH on the > configuration plus a dedicated attach privilege on the namespace or > table), > all granted to catalog roles. It shouldn't be TABLE_WRITE_PROPERTIES, > because TABLE_WRITE_DATA includes it and every writer could move a > table to > another identity. I'd leave out detach. Unlike a policy, unbinding a > table > that has data only points it at a configuration that can't reach its > files. > Changing a binding is a rebind, with the same location check as above. > > On your questions: > > 1. > > Skipping Phase 1: +1. > 2. > > Option B. Policies already do most of what we need, but they fit storage > poorly. A policy lives under a namespace, and POLICY_CREATE is a > namespace > privilege. Its content is an untyped JSON string. Applicable policies > are > returned to anyone with TABLE_READ_PROPERTIES, so role ARNs and external > IDs would be visible to every table reader. And resolving them costs a > lookup per path level on every loadTable. I'd copy the patterns instead > (attach checked on both sides, versioned updates) into a STORAGE_CONFIG > entity, and keep the binding as a configuration name on the namespace or > table, as you and Dmitri suggested in June. > 3. > > Scope. Catalog-scoped. A configuration holds an identity (role ARN and > external ID, Azure tenant, GCS service account) and a fence (allowed > locations, base location). The identity is the part worth sharing. The > fence has to stay per catalog, because the cross-catalog overlap check > is > what stops one catalog's admin from getting credentials for another > catalog's files. A configuration shared across catalogs, locations > included, would fail that check, and the only way around it is > ALLOW_OVERLAPPING_CATALOG_URLS, which is realm-wide. So I'd start > catalog-scoped with the identity inline. Later the identity can become a > realm-level object, managed by service_admin, that configurations > reference > by name. That's additive, and close to how Unity Catalog separates > credentials from external locations. > > One more thing worth changing while we're at it: how a table ends up on a > configuration other than its parent's. A binding needs an existing table, > but > Polaris writes the first metadata.json (and, for CTAS, vends the first > credentials) with the parent's configuration. A Polaris-specific > create-and-bind > call would get around, but Spark, Trino and dbt can't use it directly. > > Instead (or in addition), an admin could bind a name ahead of time: > "finance.ledger > uses configuration finance". When a client then creates finance.ledger > through the normal createTable, Polaris uses that configuration for the > location, the first write and the credentials, and moves the binding onto > the > new table. Namespaces work the same way, which is what fixes point 1. The > obvious concerns are stale bindings, renames, drop and re-create, and who > owns a name that doesn't exist yet. I think each has an answer: a pending > binding is used up at create and from then on behaves like any other > binding, dropping the configuration deletes its pending bindings, and the > namespace owns the names under it, so binding ahead needs the same > privileges as binding the namespace. It does need a small table for pending > bindings. If that's too much, we could allow it for namespaces only; tables > then follow their namespace. > > If this sounds right, I'll update the design doc and share it here before > opening a spec PR. I'll keep #5721 open until then. >
