I’m okay if there are strong and common use cases that genuinely require a catalog-level ALLOW_SETTING_S3_ENDPOINTS. I also agree with Eric that the separation between catalog config and storage config isn’t always straightforward.
That said, to avoid premature optimization, I suggest we first validate whether per-catalog ALLOW_SETTING_S3_ENDPOINTS usage is actually common or just theoretical. If such cases are rare, they might be better addressed by organizing those catalogs under separate realms, rather than introducing new per-catalog configuration semantics that complicate the governance model. Yufei On Tue, Oct 14, 2025 at 12:50 PM Eric Maynard <[email protected]> wrote: > I agree that this is maybe a weird example but I think it’s a valid one. If > I understand Dmitri’s point correctly, he’s highlighting the strange > relationship between catalogs (and their configs) and storage > configurations. > > We have a catalog config that constrains how you can configure the storage > config attached to a catalog. Yet, we require that catalogs have a storage > config present at creation time. So should the storage config defer to the > catalog config? What if we had a check that went the other way — a catalog > config that is only valid to set if the storage configuration on a catalog > is set up a certain way (e.g. on a particular cloud)? > > In general, my view is that there will be value in *decoupling* catalogs > from storage configs. This would allow you to create a (unusable?) catalog > without a storage config and set some catalog configs, which could then > control what kind of storage config can be attached to the catalog. You > could maybe attach a given storage config to multiple catalogs. This would > also clarify the “direction” of the dependency between catalogs and storage > configs. > > This decoupling is also useful if we ever want to be able to set storage > configs on any level other than catalog (e.g. different storage configs for > different namespaces could allow you to implement a multi-cloud catalog). > Indeed, multi-bucket tables are a longstanding feature request. > > However I don’t think this decoupling is easy to implement. It has > implications on persistence and the privilege model, for one thing. We’d > also have to implement some mechanism for resolving the storage config that > applies to an arbitrary entity. For just the sake of > ALLOW_SETTING_S3_ENDPOINTS, > it may not yet be necessary imo. > > —EM > > > On Tue, Oct 14, 2025 at 11:33 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Yufei, > > > > The ALLOW_SETTING_S3_ENDPOINTS flag affects catalogs. > > > > Is it not natural to allow different catalogs to have different values > for > > this flag? > > > > ... especially considering that there is config method [1] for checking > > these flags with the following parameters: > > > > getConfig(PolarisConfiguration<T> config, CatalogEntity catalogEntity) > > > > This discussion is not so much about supporting any particular use case > in > > Polaris, but mostly about making Polaris internally consistent. > > > > [1] > > > > > https://github.com/apache/polaris/blob/de351deb5648d4785639b0150d0c9df2aaab5b98/polaris-core/src/main/java/org/apache/polaris/core/config/RealmConfig.java#L65 > > > > Thanks, > > Dmitri. > > > > On Tue, Oct 14, 2025 at 2:22 PM Yufei Gu <[email protected]> wrote: > > > > > My question is: why should we set ALLOW_SETTING_S3_ENDPOINTS at the > > catalog > > > level in the first place? It was intentionally designed as a per-realm > > > setting to support specifying custom S3 endpoints during catalog > > creation. > > > > > > For reference, here’s the current description of the config: > > > > > > .key("ALLOW_SETTING_S3_ENDPOINTS") > > > .description( > > > "If set to true (default), Polaris will permit S3 storage > > > configurations to have custom endpoints.\n" > > > + "If set to false, Polaris will not accept catalog create and > update > > > requests that contain \n" > > > + "S3 endpoint properties.") > > > > > > > > > Could you help clarify the use case for pushing this down to individual > > > catalogs? > > > > > > Yufei > > > > > > > > > On Tue, Oct 14, 2025 at 11:15 AM Dmitri Bourlatchkov <[email protected] > > > > > wrote: > > > > > > > Hi Yufei, > > > > > > > > > Hi Dmitri, could you elaborate a bit more on the intended use case? > > > > > > > > Please see > > > > https://lists.apache.org/thread/4mxk3p9qdsjmty9yqyrjoow522722r2s > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Tue, Oct 14, 2025 at 1:51 PM Yufei Gu <[email protected]> > wrote: > > > > > > > > > I'm also not sure ALLOW_SETTING_S3_ENDPOINTS is the best example to > > > > > illustrate the chicken-and-egg issue. It feels like this should be > a > > > > > per-realm config, rather than something pushed down to the catalog > > > level. > > > > > > > > > > Hi Dmitri, could you elaborate a bit more on the intended use case? > > Or, > > > > if > > > > > possible, share another configuration example that better > highlights > > > the > > > > > issue? > > > > > > > > > > Yufei > > > > > > > > > > > > > > > On Tue, Oct 14, 2025 at 10:28 AM Michael Collado < > > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > Shouldn't catalog creators be able to specify some of these > > overrides > > > > by > > > > > > setting properties on the catalog itself? I see that > > > > > > ALLOW_SETTING_S3_ENDPOINTS in particular doesn't have a > > catalog-level > > > > > > configuration key, but many catalogs do. I can imagine a > > > service-level > > > > > > configuration for ALLOW_SETTING_S3_ENDPOINTS with a catalog-level > > key > > > > > that > > > > > > can be set by the catalog creator at creation time. Does that > solve > > > > > > the problem? > > > > > > > > > > > > Mike > > > > > > > > > > > > On Tue, Oct 14, 2025 at 8:44 AM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > To clarify my earlier email: Certain call paths require the > > storage > > > > > > config > > > > > > > to be provided at catalog creation time, IIRC. At the same time > > > > > > processing > > > > > > > storage config requires access to feature flags. > > > > > > > > > > > > > > Admins can indeed manage grants without exposing access to end > > > users. > > > > > > > However, I think the chicken and egg problem still exists with > > > > > > > storage configuration even for admins. > > > > > > > > > > > > > > Cheers, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Tue, Oct 14, 2025 at 1:48 AM Eric Maynard < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Could an administrator implement this two-step process by > first > > > > > > creating > > > > > > > > the catalog and granting themself " CATALOG_MANAGE_CONTENT " > > > > before > > > > > > > doing > > > > > > > > any other grants? > > > > > > > > > > > > > > > > --EM > > > > > > > > > > > > > > > > On Mon, Oct 13, 2025 at 10:25 AM Jean-Baptiste Onofré < > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Dmitri > > > > > > > > > > > > > > > > > > That's a good point. > > > > > > > > > Imho, we should have a two step approach for catalog > > creation: > > > > > first > > > > > > > > > create the "abstract" entity, and then all permission, etc. > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > JB > > > > > > > > > > > > > > > > > > On Thu, Sep 25, 2025 at 5:36 PM Dmitri Bourlatchkov < > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > Our feature flags code supports setting flags per catalog > > > [1]. > > > > > > > However > > > > > > > > > when > > > > > > > > > > dealing with catalog creation, it may be necessary to > check > > > > those > > > > > > > flags > > > > > > > > > too. > > > > > > > > > > > > > > > > > > > > This creates a chicken and egg problem where certain > flags > > > that > > > > > > apply > > > > > > > > to > > > > > > > > > > catalogs (e.g. ALLOW_SETTING_S3_ENDPOINTS) can only be > set > > > per > > > > > > realm. > > > > > > > > > > > > > > > > > > > > Would it make sense to allow a two phase approach to > > creating > > > > > > > catalogs > > > > > > > > > > where > > > > > > > > > > 1) a catalog object is created as an empty shell (ID + > > name) > > > > > > > > > > 2) An admin user adjusts feature flags / permissions > > > > > > > > > > 3) A regular user sets catalog config properties > > > > > > > > > > > > > > > > > > > > Any other thoughts / suggestions on this matter? > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/453e9fb19aaad48f8c46ef4ffe3d516df62e4706/polaris-core/src/main/java/org/apache/polaris/core/config/PolarisConfiguration.java#L167 > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
