Hi Yufei, The ALLOW_SETTING_S3_ENDPOINTS flag affects catalogs.
Is it not natural to allow different catalogs to have different values for this flag? ... especially considering that there is config method [1] for checking these flags with the following parameters: getConfig(PolarisConfiguration<T> config, CatalogEntity catalogEntity) This discussion is not so much about supporting any particular use case in Polaris, but mostly about making Polaris internally consistent. [1] https://github.com/apache/polaris/blob/de351deb5648d4785639b0150d0c9df2aaab5b98/polaris-core/src/main/java/org/apache/polaris/core/config/RealmConfig.java#L65 Thanks, Dmitri. On Tue, Oct 14, 2025 at 2:22 PM Yufei Gu <[email protected]> wrote: > My question is: why should we set ALLOW_SETTING_S3_ENDPOINTS at the catalog > level in the first place? It was intentionally designed as a per-realm > setting to support specifying custom S3 endpoints during catalog creation. > > For reference, here’s the current description of the config: > > .key("ALLOW_SETTING_S3_ENDPOINTS") > .description( > "If set to true (default), Polaris will permit S3 storage > configurations to have custom endpoints.\n" > + "If set to false, Polaris will not accept catalog create and update > requests that contain \n" > + "S3 endpoint properties.") > > > Could you help clarify the use case for pushing this down to individual > catalogs? > > Yufei > > > On Tue, Oct 14, 2025 at 11:15 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Yufei, > > > > > Hi Dmitri, could you elaborate a bit more on the intended use case? > > > > Please see > > https://lists.apache.org/thread/4mxk3p9qdsjmty9yqyrjoow522722r2s > > > > Cheers, > > Dmitri. > > > > On Tue, Oct 14, 2025 at 1:51 PM Yufei Gu <[email protected]> wrote: > > > > > I'm also not sure ALLOW_SETTING_S3_ENDPOINTS is the best example to > > > illustrate the chicken-and-egg issue. It feels like this should be a > > > per-realm config, rather than something pushed down to the catalog > level. > > > > > > Hi Dmitri, could you elaborate a bit more on the intended use case? Or, > > if > > > possible, share another configuration example that better highlights > the > > > issue? > > > > > > Yufei > > > > > > > > > On Tue, Oct 14, 2025 at 10:28 AM Michael Collado < > [email protected] > > > > > > wrote: > > > > > > > Shouldn't catalog creators be able to specify some of these overrides > > by > > > > setting properties on the catalog itself? I see that > > > > ALLOW_SETTING_S3_ENDPOINTS in particular doesn't have a catalog-level > > > > configuration key, but many catalogs do. I can imagine a > service-level > > > > configuration for ALLOW_SETTING_S3_ENDPOINTS with a catalog-level key > > > that > > > > can be set by the catalog creator at creation time. Does that solve > > > > the problem? > > > > > > > > Mike > > > > > > > > On Tue, Oct 14, 2025 at 8:44 AM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > To clarify my earlier email: Certain call paths require the storage > > > > config > > > > > to be provided at catalog creation time, IIRC. At the same time > > > > processing > > > > > storage config requires access to feature flags. > > > > > > > > > > Admins can indeed manage grants without exposing access to end > users. > > > > > However, I think the chicken and egg problem still exists with > > > > > storage configuration even for admins. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Tue, Oct 14, 2025 at 1:48 AM Eric Maynard < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Could an administrator implement this two-step process by first > > > > creating > > > > > > the catalog and granting themself " CATALOG_MANAGE_CONTENT " > > before > > > > > doing > > > > > > any other grants? > > > > > > > > > > > > --EM > > > > > > > > > > > > On Mon, Oct 13, 2025 at 10:25 AM Jean-Baptiste Onofré < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Dmitri > > > > > > > > > > > > > > That's a good point. > > > > > > > Imho, we should have a two step approach for catalog creation: > > > first > > > > > > > create the "abstract" entity, and then all permission, etc. > > > > > > > > > > > > > > Regards > > > > > > > JB > > > > > > > > > > > > > > On Thu, Sep 25, 2025 at 5:36 PM Dmitri Bourlatchkov < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > Our feature flags code supports setting flags per catalog > [1]. > > > > > However > > > > > > > when > > > > > > > > dealing with catalog creation, it may be necessary to check > > those > > > > > flags > > > > > > > too. > > > > > > > > > > > > > > > > This creates a chicken and egg problem where certain flags > that > > > > apply > > > > > > to > > > > > > > > catalogs (e.g. ALLOW_SETTING_S3_ENDPOINTS) can only be set > per > > > > realm. > > > > > > > > > > > > > > > > Would it make sense to allow a two phase approach to creating > > > > > catalogs > > > > > > > > where > > > > > > > > 1) a catalog object is created as an empty shell (ID + name) > > > > > > > > 2) An admin user adjusts feature flags / permissions > > > > > > > > 3) A regular user sets catalog config properties > > > > > > > > > > > > > > > > Any other thoughts / suggestions on this matter? > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/453e9fb19aaad48f8c46ef4ffe3d516df62e4706/polaris-core/src/main/java/org/apache/polaris/core/config/PolarisConfiguration.java#L167 > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > >
