My question is: why should we set ALLOW_SETTING_S3_ENDPOINTS at the catalog
level in the first place? It was intentionally designed as a per-realm
setting to support specifying custom S3 endpoints during catalog creation.
For reference, here’s the current description of the config:
.key("ALLOW_SETTING_S3_ENDPOINTS")
.description(
"If set to true (default), Polaris will permit S3 storage
configurations to have custom endpoints.\n"
+ "If set to false, Polaris will not accept catalog create and update
requests that contain \n"
+ "S3 endpoint properties.")
Could you help clarify the use case for pushing this down to individual
catalogs?
Yufei
On Tue, Oct 14, 2025 at 11:15 AM Dmitri Bourlatchkov <[email protected]>
wrote:
> Hi Yufei,
>
> > Hi Dmitri, could you elaborate a bit more on the intended use case?
>
> Please see
> https://lists.apache.org/thread/4mxk3p9qdsjmty9yqyrjoow522722r2s
>
> Cheers,
> Dmitri.
>
> On Tue, Oct 14, 2025 at 1:51 PM Yufei Gu <[email protected]> wrote:
>
> > I'm also not sure ALLOW_SETTING_S3_ENDPOINTS is the best example to
> > illustrate the chicken-and-egg issue. It feels like this should be a
> > per-realm config, rather than something pushed down to the catalog level.
> >
> > Hi Dmitri, could you elaborate a bit more on the intended use case? Or,
> if
> > possible, share another configuration example that better highlights the
> > issue?
> >
> > Yufei
> >
> >
> > On Tue, Oct 14, 2025 at 10:28 AM Michael Collado <[email protected]
> >
> > wrote:
> >
> > > Shouldn't catalog creators be able to specify some of these overrides
> by
> > > setting properties on the catalog itself? I see that
> > > ALLOW_SETTING_S3_ENDPOINTS in particular doesn't have a catalog-level
> > > configuration key, but many catalogs do. I can imagine a service-level
> > > configuration for ALLOW_SETTING_S3_ENDPOINTS with a catalog-level key
> > that
> > > can be set by the catalog creator at creation time. Does that solve
> > > the problem?
> > >
> > > Mike
> > >
> > > On Tue, Oct 14, 2025 at 8:44 AM Dmitri Bourlatchkov <[email protected]>
> > > wrote:
> > >
> > > > To clarify my earlier email: Certain call paths require the storage
> > > config
> > > > to be provided at catalog creation time, IIRC. At the same time
> > > processing
> > > > storage config requires access to feature flags.
> > > >
> > > > Admins can indeed manage grants without exposing access to end users.
> > > > However, I think the chicken and egg problem still exists with
> > > > storage configuration even for admins.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Tue, Oct 14, 2025 at 1:48 AM Eric Maynard <
> [email protected]
> > >
> > > > wrote:
> > > >
> > > > > Could an administrator implement this two-step process by first
> > > creating
> > > > > the catalog and granting themself " CATALOG_MANAGE_CONTENT "
> before
> > > > doing
> > > > > any other grants?
> > > > >
> > > > > --EM
> > > > >
> > > > > On Mon, Oct 13, 2025 at 10:25 AM Jean-Baptiste Onofré <
> > [email protected]
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Dmitri
> > > > > >
> > > > > > That's a good point.
> > > > > > Imho, we should have a two step approach for catalog creation:
> > first
> > > > > > create the "abstract" entity, and then all permission, etc.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > On Thu, Sep 25, 2025 at 5:36 PM Dmitri Bourlatchkov <
> > > [email protected]>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Our feature flags code supports setting flags per catalog [1].
> > > > However
> > > > > > when
> > > > > > > dealing with catalog creation, it may be necessary to check
> those
> > > > flags
> > > > > > too.
> > > > > > >
> > > > > > > This creates a chicken and egg problem where certain flags that
> > > apply
> > > > > to
> > > > > > > catalogs (e.g. ALLOW_SETTING_S3_ENDPOINTS) can only be set per
> > > realm.
> > > > > > >
> > > > > > > Would it make sense to allow a two phase approach to creating
> > > > catalogs
> > > > > > > where
> > > > > > > 1) a catalog object is created as an empty shell (ID + name)
> > > > > > > 2) An admin user adjusts feature flags / permissions
> > > > > > > 3) A regular user sets catalog config properties
> > > > > > >
> > > > > > > Any other thoughts / suggestions on this matter?
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/polaris/blob/453e9fb19aaad48f8c46ef4ffe3d516df62e4706/polaris-core/src/main/java/org/apache/polaris/core/config/PolarisConfiguration.java#L167
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > >
> > > > >
> > > >
> > >
> >
>
