> Yet, no one answered my second question from the initial email. That is:
why do we need to expose non-IRC endpoints in the IRC config response in
the first place?
Apologies if I didn't explain this clearly in my earlier email.
With Polaris, you can use a pure IRC client, or you can use a Polaris
client that leverages Polaris specific capabilities such as generic tables
and policies in addition to IRC functionality. The community has already
decided to reuse the IRC config endpoint for capability discovery. I shared
an example from the Polaris Spark client previously, but I'll repeat it
here because it illustrates the point well. Note that the client supports
both Iceberg tables and generic tables:
public List<TableIdentifier> listGenericTables(Namespace ns) {
Endpoint.check(endpoints, PolarisEndpoints.V1_LIST_GENERIC_TABLES);
}
In this model, the config endpoint acts as a capability discovery
mechanism. Standard IRC clients can simply ignore endpoints they do not
recognize, while Polaris aware clients can use the additional information
to enable Polaris specific functionality.
I also think Adam made a good point. The /api/catalog/v1/config endpoint
returns both IRC and Polaris capabilities. Otherwise, we need to introduce
a separate Polaris-specific discovery endpoint, requiring clients to query
and manage two different capability sources.
Yufei
On Mon, Jun 22, 2026 at 5:28 PM Adnan Hemani via dev <[email protected]>
wrote:
> Ah yes, my mistake. But I think a similar point exists: Does the service
> need to know about all new endpoint information? Can we not make it dumb
> and gate on a server-wide feature flag or something similar?
>
> -Adnan
>
> On Mon, Jun 22, 2026 at 5:02 PM Dmitri Bourlatchkov <[email protected]>
> wrote:
>
> > Hi Adnan,
> >
> > You might have misinterpreted my message. My point was about the server
> > (service) side, not client side.
> >
> > Cheers,
> > Dmitri.
> >
> > On Mon, Jun 22, 2026 at 7:58 PM Adnan Hemani via dev <
> > [email protected]>
> > wrote:
> >
> > > > For example, if we are to expose Open Lineage or OSI endpoints in IRC
> > > config, the IRC service code will have to be aware of OL / OSI modules.
> > >
> > > I don't agree with this statement; the IRC client is always free to
> > > disregard any of these endpoints/modules. Returning this information
> does
> > > not force a client to take any action based on this extra endpoint
> > > information.
> > >
> > > I do agree that if this forced the IRC client to be fundamentally
> altered
> > > because of these additional endpoints, then that would be a significant
> > > issue. But as Russell stated above, "what would actually break if
> someone
> > > interpreted the spec this way?"
> > >
> > > Best,
> > > Adnan Hemani
> > >
> > > On Mon, Jun 22, 2026 at 3:30 PM Dmitri Bourlatchkov <[email protected]>
> > > wrote:
> > >
> > > > Hi Russell,
> > > >
> > > > Your points sound reasonable to me.
> > > >
> > > > Yet, noone answered my second question from my initial email. That
> is:
> > > why
> > > > do we need to expose non-IRC endpoints in the IRC config response in
> > the
> > > > first place?
> > > >
> > > > Regarding the maintainability aspect, I see this becoming a more
> > > > substantial concern if it evolves into a self-sustaining pattern.
> > > >
> > > > For example, if we are to expose Open Lineage or OSI endpoints in IRC
> > > > config, the IRC service code will have to be aware of OL / OSI
> > modules. I
> > > > think this dependency is conceptually incorrect.
> > > >
> > > > I do not so much mind existing non-IRC endpoints in the IRC config,
> but
> > > > extending this to new APIs feels like an anti-pattern to me.
> > > >
> > > > Cheers,
> > > > Dmitri.
> > > >
> > > > On Mon, Jun 22, 2026 at 5:46 PM Russell Spitzer <
> > > [email protected]
> > > > >
> > > > wrote:
> > > >
> > > > > From a very pragmatic point of view, I'm not sure what we'd gain
> from
> > > > > separate
> > > > > configuration mechanisms. We already have one and it's worked well
> so
> > > > far,
> > > > > do we have a compelling reason to swap right now?
> > > > >
> > > > > As best I can tell, the worst case is that the IRC spec later
> > tightens
> > > > the
> > > > > semantics
> > > > > of the endpoints field, making our extra entries non-compliant. In
> > that
> > > > > situation
> > > > > we'd have to change our response, add a Polaris-specific
> > > config/discovery
> > > > > endpoint, and
> > > > > have our clients move over to it. But that's exactly the work this
> > > > proposal
> > > > > is asking us to do today
> > > > > so we would be taking same burden now instead of potentially never.
> > I'd
> > > > > rather defer
> > > > > it until something concrete forces our hand.
> > > > >
> > > > > The one counter I can see is that decoupling gets harder as more
> > > clients
> > > > > adopt it. But
> > > > > since the Polaris endpoints are already namespaced (/polaris/...),
> > > > lifting
> > > > > them into a separate endpoint
> > > > > later stays mechanical, so I don't think we're meaningfully
> cheaper
> > > > doing
> > > > > it now.
> > > > >
> > > > >
> > > > > I'd like to gently point out that it doesn't feel especially
> > important
> > > > > whether the exact wording
> > > > > or original intent of the spec supports what we're doing. The
> > > discussion
> > > > is
> > > > > starting to feel
> > > > > like a legalistic reading of the spec rather than a focus on
> concrete
> > > > > impact. I'd suggest we focus
> > > > > on arguments like "what would actually break if someone interpreted
> > the
> > > > > spec this way?"
> > > > > rather than "what was the original intent of the text?"
> > > > >
> > > > > On Mon, Jun 22, 2026 at 3:23 PM Adnan Hemani via dev <
> > > > > [email protected]>
> > > > > wrote:
> > > > >
> > > > > > Hi Dmitri,
> > > > > >
> > > > > > I think you and I view the same statement very differently - I
> > > > definitely
> > > > > > don't think it's "pretty clear that the scope of the data
> returned
> > > > under
> > > > > > that type is limited to the API defined by the spec". I see it
> this
> > > > way:
> > > > > if
> > > > > > this statement truly meant ONLY IRC endpoints, the spec would
> > > > explicitly
> > > > > > use the words "IRC spec endpoints". The absence of clarification
> in
> > > any
> > > > > > specification rarely means you should "read-between-the-lines" to
> > > > assume
> > > > > > the writer's "intent".
> > > > > >
> > > > > > I will look into Iceberg mailing list threads and PRs for
> > background
> > > > > > information to clarify things.
> > > > > >
> > > > > > Best,
> > > > > > Adnan Hemani
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 22, 2026 at 12:27 PM Dmitri Bourlatchkov <
> > > [email protected]
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > HI Adnan,
> > > > > > >
> > > > > > > The IRC spec defines a "type" for the config response. A
> property
> > > of
> > > > > that
> > > > > > > "type" is a list of "endpoints supported by the server". There
> is
> > > no
> > > > > > > statement about generalizing those endpoints to non-IRC APIs.
> > > > > > >
> > > > > > > Therefore, I think it is pretty clear that the scope of the
> data
> > > > > returned
> > > > > > > under that type is limited to the API defined by the spec,
> which
> > is
> > > > > IRC.
> > > > > > >
> > > > > > > Let's expand this discussion a bit.
> > > > > > >
> > > > > > > What is the rationale for returning non-IRC endpoints in the
> IRC
> > > > config
> > > > > > > response?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Dmitri.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jun 22, 2026 at 3:21 PM Adnan Hemani via dev <
> > > > > > > [email protected]> wrote:
> > > > > > >
> > > > > > >> Hi Dmitri,
> > > > > > >>
> > > > > > >> I'm not sure where you are assuming that "endpoints" only
> refers
> > > to
> > > > > > >> Iceberg
> > > > > > >> endpoints in the IRC spec. Do you have any thing you can point
> > us
> > > to
> > > > > > >> regarding this?
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Adnan Hemani
> > > > > > >>
> > > > > > >> On Mon, Jun 22, 2026 at 12:18 PM Dmitri Bourlatchkov <
> > > > > [email protected]>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi Adam,
> > > > > > >> >
> > > > > > >> > > #1 - Is GET '
> > > > > > >> >
> http://localhost:8181/api/catalog/v1/config?warehouse=polaris
> > '
> > > > only
> > > > > > >> > an Iceberg Catalog concept?
> > > > > > >> >
> > > > > > >> > My interpretation is YES, because the payload of that
> response
> > > is
> > > > > > >> defined
> > > > > > >> > by the IRC API spec [1].
> > > > > > >> >
> > > > > > >> > I believe one can reasonably assume that the statement
> > > "endpoints
> > > > > > >> supported
> > > > > > >> > by the server" is scoped only to the IRC API itself. There's
> > no
> > > > > > >> provision
> > > > > > >> > in that spec about covering all possible APIs.
> > > > > > >> >
> > > > > > >> > That said, I'd be happy to discuss how clients can discover
> > > > Polaris
> > > > > > >> > features in general without overloading existing
> > specifications.
> > > > > > >> >
> > > > > > >> > [1]
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/open-api/rest-catalog-open-api.yaml#L105
> > > > > > >> >
> > > > > > >> > On Mon, Jun 22, 2026 at 2:52 PM Adam Christian <
> > > > > > >> > [email protected]> wrote:
> > > > > > >> >
> > > > > > >> > > I agree with both Yufei & Dmitri:
> > > > > > >> > >
> > > > > > >> > > #1 - It seems like improper coupling for an Iceberg REST
> > > Catalog
> > > > > > >> config
> > > > > > >> > to
> > > > > > >> > > return a configuration unrelated to Iceberg.
> > > > > > >> > > #2 - Polaris is a superset of an Iceberg REST Catalog.
> > > > > > >> > >
> > > > > > >> > > So, in my opinion, the question is more aptly framed as
> two
> > > > > > questions:
> > > > > > >> > >
> > > > > > >> > > #1 - Is GET '
> > > > > > >> >
> http://localhost:8181/api/catalog/v1/config?warehouse=polaris
> > > > > > >> > > '
> > > > > > >> > > only an Iceberg Catalog concept?
> > > > > > >> > > #2 - Should GET '
> > > > > > >> > >
> > http://localhost:8181/api/catalog/v1/config?warehouse=polaris
> > > '
> > > > > > return
> > > > > > >> > > additional endpoints?
> > > > > > >> > >
> > > > > > >> > > I believe that the answer to #1 is no, but that requires
> > > changes
> > > > > to
> > > > > > >> the
> > > > > > >> > > codebase. I believe the answer to #2 is yes.
> > > > > > >> > >
> > > > > > >> > > Based on the codebase, we are exposing several endpoints
> in
> > > this
> > > > > > >> > > configuration API which are not Iceberg endpoints. For
> > > example,
> > > > in
> > > > > > the
> > > > > > >> > > Generic Tables case, it is explicitly not Iceberg. In my
> > > > opinion,
> > > > > > >> this is
> > > > > > >> > > alright because our API specification only says
> > "Configuration
> > > > > API,"
> > > > > > >> not
> > > > > > >> > > "Iceberg Configuration API." Yes, it is
> Iceberg-compatible,
> > > but
> > > > it
> > > > > > is
> > > > > > >> not
> > > > > > >> > > Iceberg-centric. This retrieves the configuration for the
> > > > Catalog;
> > > > > > >> > > regardless of whether it is an Iceberg Catalog. Now, if
> this
> > > is
> > > > > > truly
> > > > > > >> a
> > > > > > >> > > superset, it implies that IcebergCatalogHandler.java
> should
> > > not
> > > > > > handle
> > > > > > >> > > returning the configuration. Additionally, we need to
> > return a
> > > > > > >> superset
> > > > > > >> > of
> > > > > > >> > > the ConfigResponse (an Iceberg-core concept) for the REST
> > API
> > > > > > >> response.
> > > > > > >> > >
> > > > > > >> > > Now, if the answer to #1 is yes, that means we should
> > probably
> > > > > have
> > > > > > a
> > > > > > >> > > separate API for configuration.
> > > > > > >> > >
> > > > > > >> > > What do y'all think?
> > > > > > >> > >
> > > > > > >> > > Go community,
> > > > > > >> > >
> > > > > > >> > > Adam
> > > > > > >> > >
> > > > > > >> > > On Mon, Jun 22, 2026 at 1:55 PM Yufei Gu <
> > > [email protected]>
> > > > > > >> wrote:
> > > > > > >> > >
> > > > > > >> > > > I think a Polaris client is also an IRC client, just
> with
> > > > > > additional
> > > > > > >> > > > capabilities. In that sense, Polaris can be viewed as a
> > > > superset
> > > > > > of
> > > > > > >> > IRC.
> > > > > > >> > > If
> > > > > > >> > > > the config endpoint is intended for capability
> discovery,
> > > > > > returning
> > > > > > >> > > Polaris
> > > > > > >> > > > specific endpoints seems reasonable. Standard IRC
> clients
> > > can
> > > > > > simply
> > > > > > >> > > ignore
> > > > > > >> > > > endpoints they don't recognize.
> > > > > > >> > > >
> > > > > > >> > > > A concrete example is the Polaris Spark client, which
> > checks
> > > > > > server
> > > > > > >> > > > capabilities before using Polaris specific
> functionality:
> > > > > > >> > > >
> > > > > > >> > > > public List<TableIdentifier> listGenericTables(Namespace
> > > ns) {
> > > > > > >> > > > Endpoint.check(endpoints,
> > > > > > >> PolarisEndpoints.V1_LIST_GENERIC_TABLES);
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > Yufei
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Fri, Jun 19, 2026 at 7:43 AM Dmitri Bourlatchkov <
> > > > > > >> [email protected]>
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi All,
> > > > > > >> > > > >
> > > > > > >> > > > > During my review of [4816] I realized [1] that Polaris
> > > > returns
> > > > > > >> some
> > > > > > >> > > > non-IRC
> > > > > > >> > > > > endpoints in the IRC config responses.
> > > > > > >> > > > >
> > > > > > >> > > > > For example:
> > > > > > >> > > > >
> > > > > > >> > > > > GET '
> > > > > > >> http://localhost:8181/api/catalog/v1/config?warehouse=polaris
> '
> > > > > > >> > > > >
> > > > > > >> > > > > Result:
> > > > > > >> > > > > [...]
> > > > > > >> > > > > "endpoints": [
> > > > > > >> > > > > "GET /v1/{prefix}/namespaces",
> > > > > > >> > > > > "GET /v1/{prefix}/namespaces/{namespace}",
> > > > > > >> > > > > "HEAD /v1/{prefix}/namespaces/{namespace}",
> > > > > > >> > > > > [...]
> > > > > > >> > > > > "DELETE
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> polaris/v1/{prefix}/namespaces/{namespace}/generic-tables/{generic-table}",
> > > > > > >> > > > > "GET
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> polaris/v1/{prefix}/namespaces/{namespace}/generic-tables/{generic-table}",
> > > > > > >> > > > > "GET
> > > > > /polaris/v1/{prefix}/namespaces/{namespace}/policies",
> > > > > > >> > > > > "POST
> > > > > /polaris/v1/{prefix}/namespaces/{namespace}/policies",
> > > > > > >> > > > >
> > > > > > >> > > > > The latter group of endpoints is not related to the
> > > Iceberg
> > > > > REST
> > > > > > >> > > Catalog
> > > > > > >> > > > > API.
> > > > > > >> > > > >
> > > > > > >> > > > > I wonder what the rationale might be for returning
> them
> > in
> > > > the
> > > > > > IRC
> > > > > > >> > > config
> > > > > > >> > > > > response.
> > > > > > >> > > > >
> > > > > > >> > > > > From my POV, returning them in the IRC config response
> > is
> > > > > > >> incorrect
> > > > > > >> > > > because
> > > > > > >> > > > > these endpoints form APIs that follow a different
> > > > > specification
> > > > > > >> and
> > > > > > >> > > > clients
> > > > > > >> > > > > using the IRC API do not need that information to
> > properly
> > > > use
> > > > > > the
> > > > > > >> > IRC
> > > > > > >> > > > API.
> > > > > > >> > > > >
> > > > > > >> > > > > WDYT?
> > > > > > >> > > > >
> > > > > > >> > > > > [1]
> > > > > > >> >
> > > > https://github.com/apache/polaris/pull/4816#discussion_r3438945230
> > > > > > >> > > > >
> > > > > > >> > > > > [4816] https://github.com/apache/polaris/pull/4816
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Dmitri.
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
