I agree that this is maybe a weird example but I think it’s a valid one. If I understand Dmitri’s point correctly, he’s highlighting the strange relationship between catalogs (and their configs) and storage configurations.
We have a catalog config that constrains how you can configure the storage config attached to a catalog. Yet, we require that catalogs have a storage config present at creation time. So should the storage config defer to the catalog config? What if we had a check that went the other way — a catalog config that is only valid to set if the storage configuration on a catalog is set up a certain way (e.g. on a particular cloud)? In general, my view is that there will be value in *decoupling* catalogs from storage configs. This would allow you to create a (unusable?) catalog without a storage config and set some catalog configs, which could then control what kind of storage config can be attached to the catalog. You could maybe attach a given storage config to multiple catalogs. This would also clarify the “direction” of the dependency between catalogs and storage configs. This decoupling is also useful if we ever want to be able to set storage configs on any level other than catalog (e.g. different storage configs for different namespaces could allow you to implement a multi-cloud catalog). Indeed, multi-bucket tables are a longstanding feature request. However I don’t think this decoupling is easy to implement. It has implications on persistence and the privilege model, for one thing. We’d also have to implement some mechanism for resolving the storage config that applies to an arbitrary entity. For just the sake of ALLOW_SETTING_S3_ENDPOINTS, it may not yet be necessary imo. —EM On Tue, Oct 14, 2025 at 11:33 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Yufei, > > The ALLOW_SETTING_S3_ENDPOINTS flag affects catalogs. > > Is it not natural to allow different catalogs to have different values for > this flag? > > ... especially considering that there is config method [1] for checking > these flags with the following parameters: > > getConfig(PolarisConfiguration<T> config, CatalogEntity catalogEntity) > > This discussion is not so much about supporting any particular use case in > Polaris, but mostly about making Polaris internally consistent. > > [1] > > https://github.com/apache/polaris/blob/de351deb5648d4785639b0150d0c9df2aaab5b98/polaris-core/src/main/java/org/apache/polaris/core/config/RealmConfig.java#L65 > > Thanks, > Dmitri. > > On Tue, Oct 14, 2025 at 2:22 PM Yufei Gu <[email protected]> wrote: > > > My question is: why should we set ALLOW_SETTING_S3_ENDPOINTS at the > catalog > > level in the first place? It was intentionally designed as a per-realm > > setting to support specifying custom S3 endpoints during catalog > creation. > > > > For reference, here’s the current description of the config: > > > > .key("ALLOW_SETTING_S3_ENDPOINTS") > > .description( > > "If set to true (default), Polaris will permit S3 storage > > configurations to have custom endpoints.\n" > > + "If set to false, Polaris will not accept catalog create and update > > requests that contain \n" > > + "S3 endpoint properties.") > > > > > > Could you help clarify the use case for pushing this down to individual > > catalogs? > > > > Yufei > > > > > > On Tue, Oct 14, 2025 at 11:15 AM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi Yufei, > > > > > > > Hi Dmitri, could you elaborate a bit more on the intended use case? > > > > > > Please see > > > https://lists.apache.org/thread/4mxk3p9qdsjmty9yqyrjoow522722r2s > > > > > > Cheers, > > > Dmitri. > > > > > > On Tue, Oct 14, 2025 at 1:51 PM Yufei Gu <[email protected]> wrote: > > > > > > > I'm also not sure ALLOW_SETTING_S3_ENDPOINTS is the best example to > > > > illustrate the chicken-and-egg issue. It feels like this should be a > > > > per-realm config, rather than something pushed down to the catalog > > level. > > > > > > > > Hi Dmitri, could you elaborate a bit more on the intended use case? > Or, > > > if > > > > possible, share another configuration example that better highlights > > the > > > > issue? > > > > > > > > Yufei > > > > > > > > > > > > On Tue, Oct 14, 2025 at 10:28 AM Michael Collado < > > [email protected] > > > > > > > > wrote: > > > > > > > > > Shouldn't catalog creators be able to specify some of these > overrides > > > by > > > > > setting properties on the catalog itself? I see that > > > > > ALLOW_SETTING_S3_ENDPOINTS in particular doesn't have a > catalog-level > > > > > configuration key, but many catalogs do. I can imagine a > > service-level > > > > > configuration for ALLOW_SETTING_S3_ENDPOINTS with a catalog-level > key > > > > that > > > > > can be set by the catalog creator at creation time. Does that solve > > > > > the problem? > > > > > > > > > > Mike > > > > > > > > > > On Tue, Oct 14, 2025 at 8:44 AM Dmitri Bourlatchkov < > > [email protected]> > > > > > wrote: > > > > > > > > > > > To clarify my earlier email: Certain call paths require the > storage > > > > > config > > > > > > to be provided at catalog creation time, IIRC. At the same time > > > > > processing > > > > > > storage config requires access to feature flags. > > > > > > > > > > > > Admins can indeed manage grants without exposing access to end > > users. > > > > > > However, I think the chicken and egg problem still exists with > > > > > > storage configuration even for admins. > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Tue, Oct 14, 2025 at 1:48 AM Eric Maynard < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > Could an administrator implement this two-step process by first > > > > > creating > > > > > > > the catalog and granting themself " CATALOG_MANAGE_CONTENT " > > > before > > > > > > doing > > > > > > > any other grants? > > > > > > > > > > > > > > --EM > > > > > > > > > > > > > > On Mon, Oct 13, 2025 at 10:25 AM Jean-Baptiste Onofré < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Dmitri > > > > > > > > > > > > > > > > That's a good point. > > > > > > > > Imho, we should have a two step approach for catalog > creation: > > > > first > > > > > > > > create the "abstract" entity, and then all permission, etc. > > > > > > > > > > > > > > > > Regards > > > > > > > > JB > > > > > > > > > > > > > > > > On Thu, Sep 25, 2025 at 5:36 PM Dmitri Bourlatchkov < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > Our feature flags code supports setting flags per catalog > > [1]. > > > > > > However > > > > > > > > when > > > > > > > > > dealing with catalog creation, it may be necessary to check > > > those > > > > > > flags > > > > > > > > too. > > > > > > > > > > > > > > > > > > This creates a chicken and egg problem where certain flags > > that > > > > > apply > > > > > > > to > > > > > > > > > catalogs (e.g. ALLOW_SETTING_S3_ENDPOINTS) can only be set > > per > > > > > realm. > > > > > > > > > > > > > > > > > > Would it make sense to allow a two phase approach to > creating > > > > > > catalogs > > > > > > > > > where > > > > > > > > > 1) a catalog object is created as an empty shell (ID + > name) > > > > > > > > > 2) An admin user adjusts feature flags / permissions > > > > > > > > > 3) A regular user sets catalog config properties > > > > > > > > > > > > > > > > > > Any other thoughts / suggestions on this matter? > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/453e9fb19aaad48f8c46ef4ffe3d516df62e4706/polaris-core/src/main/java/org/apache/polaris/core/config/PolarisConfiguration.java#L167 > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
