Hi all, I think we should separate two questions:
1. Can non-IRC endpoints be added to the IRC config response without breaking clients? 2. What can a normal Iceberg REST client actually do with that information? The first question only tells us whether the behavior is tolerated. It does not establish that the IRC config response is the right contract for Polaris-wide capability discovery. For a normal IRC client, I do not see a concrete use case for OpenLineage, OSI, generic-table, or policy endpoints in that response. A Polaris-aware client may need discovery, but that sounds like Polaris capability discovery, not IRC capability discovery. So what is the concrete use case for exposing those endpoints to normal IRC clients? Best, Robert On Tue, Jun 23, 2026 at 3:26 PM Dmitri Bourlatchkov < [email protected]> wrote: > Hi Yufei, > > > The community has already decided to reuse the IRC config endpoint for > capability discovery > > Could you give a link to that decision? > > Thanks, > Dmitri. > > On Mon, Jun 22, 2026 at 9:45 PM Yufei Gu <[email protected]> wrote: > > > > Yet, no one answered my second question from the initial email. That > is: > > why do we need to expose non-IRC endpoints in the IRC config response in > > the first place? > > > > Apologies if I didn't explain this clearly in my earlier email. > > > > With Polaris, you can use a pure IRC client, or you can use a Polaris > > client that leverages Polaris specific capabilities such as generic > tables > > and policies in addition to IRC functionality. The community has already > > decided to reuse the IRC config endpoint for capability discovery. I > shared > > an example from the Polaris Spark client previously, but I'll repeat it > > here because it illustrates the point well. Note that the client supports > > both Iceberg tables and generic tables: > > > > public List<TableIdentifier> listGenericTables(Namespace ns) { > > Endpoint.check(endpoints, PolarisEndpoints.V1_LIST_GENERIC_TABLES); > > } > > > > In this model, the config endpoint acts as a capability discovery > > mechanism. Standard IRC clients can simply ignore endpoints they do not > > recognize, while Polaris aware clients can use the additional information > > to enable Polaris specific functionality. > > > > I also think Adam made a good point. The /api/catalog/v1/config endpoint > > returns both IRC and Polaris capabilities. Otherwise, we need to > introduce > > a separate Polaris-specific discovery endpoint, requiring clients to > query > > and manage two different capability sources. > > > > Yufei > > > > > > On Mon, Jun 22, 2026 at 5:28 PM Adnan Hemani via dev < > > [email protected]> > > wrote: > > > > > Ah yes, my mistake. But I think a similar point exists: Does the > service > > > need to know about all new endpoint information? Can we not make it > dumb > > > and gate on a server-wide feature flag or something similar? > > > > > > -Adnan > > > > > > On Mon, Jun 22, 2026 at 5:02 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi Adnan, > > > > > > > > You might have misinterpreted my message. My point was about the > server > > > > (service) side, not client side. > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Mon, Jun 22, 2026 at 7:58 PM Adnan Hemani via dev < > > > > [email protected]> > > > > wrote: > > > > > > > > > > For example, if we are to expose Open Lineage or OSI endpoints in > > IRC > > > > > config, the IRC service code will have to be aware of OL / OSI > > modules. > > > > > > > > > > I don't agree with this statement; the IRC client is always free to > > > > > disregard any of these endpoints/modules. Returning this > information > > > does > > > > > not force a client to take any action based on this extra endpoint > > > > > information. > > > > > > > > > > I do agree that if this forced the IRC client to be fundamentally > > > altered > > > > > because of these additional endpoints, then that would be a > > significant > > > > > issue. But as Russell stated above, "what would actually break if > > > someone > > > > > interpreted the spec this way?" > > > > > > > > > > Best, > > > > > Adnan Hemani > > > > > > > > > > On Mon, Jun 22, 2026 at 3:30 PM Dmitri Bourlatchkov < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi Russell, > > > > > > > > > > > > Your points sound reasonable to me. > > > > > > > > > > > > Yet, noone answered my second question from my initial email. > That > > > is: > > > > > why > > > > > > do we need to expose non-IRC endpoints in the IRC config response > > in > > > > the > > > > > > first place? > > > > > > > > > > > > Regarding the maintainability aspect, I see this becoming a more > > > > > > substantial concern if it evolves into a self-sustaining pattern. > > > > > > > > > > > > For example, if we are to expose Open Lineage or OSI endpoints in > > IRC > > > > > > config, the IRC service code will have to be aware of OL / OSI > > > > modules. I > > > > > > think this dependency is conceptually incorrect. > > > > > > > > > > > > I do not so much mind existing non-IRC endpoints in the IRC > config, > > > but > > > > > > extending this to new APIs feels like an anti-pattern to me. > > > > > > > > > > > > Cheers, > > > > > > Dmitri. > > > > > > > > > > > > On Mon, Jun 22, 2026 at 5:46 PM Russell Spitzer < > > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > From a very pragmatic point of view, I'm not sure what we'd > gain > > > from > > > > > > > separate > > > > > > > configuration mechanisms. We already have one and it's worked > > well > > > so > > > > > > far, > > > > > > > do we have a compelling reason to swap right now? > > > > > > > > > > > > > > As best I can tell, the worst case is that the IRC spec later > > > > tightens > > > > > > the > > > > > > > semantics > > > > > > > of the endpoints field, making our extra entries non-compliant. > > In > > > > that > > > > > > > situation > > > > > > > we'd have to change our response, add a Polaris-specific > > > > > config/discovery > > > > > > > endpoint, and > > > > > > > have our clients move over to it. But that's exactly the work > > this > > > > > > proposal > > > > > > > is asking us to do today > > > > > > > so we would be taking same burden now instead of potentially > > never. > > > > I'd > > > > > > > rather defer > > > > > > > it until something concrete forces our hand. > > > > > > > > > > > > > > The one counter I can see is that decoupling gets harder as > more > > > > > clients > > > > > > > adopt it. But > > > > > > > since the Polaris endpoints are already namespaced > > (/polaris/...), > > > > > > lifting > > > > > > > them into a separate endpoint > > > > > > > later stays mechanical, so I don't think we're meaningfully > > > cheaper > > > > > > doing > > > > > > > it now. > > > > > > > > > > > > > > > > > > > > > I'd like to gently point out that it doesn't feel especially > > > > important > > > > > > > whether the exact wording > > > > > > > or original intent of the spec supports what we're doing. The > > > > > discussion > > > > > > is > > > > > > > starting to feel > > > > > > > like a legalistic reading of the spec rather than a focus on > > > concrete > > > > > > > impact. I'd suggest we focus > > > > > > > on arguments like "what would actually break if someone > > interpreted > > > > the > > > > > > > spec this way?" > > > > > > > rather than "what was the original intent of the text?" > > > > > > > > > > > > > > On Mon, Jun 22, 2026 at 3:23 PM Adnan Hemani via dev < > > > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > > > > > I think you and I view the same statement very differently - > I > > > > > > definitely > > > > > > > > don't think it's "pretty clear that the scope of the data > > > returned > > > > > > under > > > > > > > > that type is limited to the API defined by the spec". I see > it > > > this > > > > > > way: > > > > > > > if > > > > > > > > this statement truly meant ONLY IRC endpoints, the spec would > > > > > > explicitly > > > > > > > > use the words "IRC spec endpoints". The absence of > > clarification > > > in > > > > > any > > > > > > > > specification rarely means you should > "read-between-the-lines" > > to > > > > > > assume > > > > > > > > the writer's "intent". > > > > > > > > > > > > > > > > I will look into Iceberg mailing list threads and PRs for > > > > background > > > > > > > > information to clarify things. > > > > > > > > > > > > > > > > Best, > > > > > > > > Adnan Hemani > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 22, 2026 at 12:27 PM Dmitri Bourlatchkov < > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > HI Adnan, > > > > > > > > > > > > > > > > > > The IRC spec defines a "type" for the config response. A > > > property > > > > > of > > > > > > > that > > > > > > > > > "type" is a list of "endpoints supported by the server". > > There > > > is > > > > > no > > > > > > > > > statement about generalizing those endpoints to non-IRC > APIs. > > > > > > > > > > > > > > > > > > Therefore, I think it is pretty clear that the scope of the > > > data > > > > > > > returned > > > > > > > > > under that type is limited to the API defined by the spec, > > > which > > > > is > > > > > > > IRC. > > > > > > > > > > > > > > > > > > Let's expand this discussion a bit. > > > > > > > > > > > > > > > > > > What is the rationale for returning non-IRC endpoints in > the > > > IRC > > > > > > config > > > > > > > > > response? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 22, 2026 at 3:21 PM Adnan Hemani via dev < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > >> Hi Dmitri, > > > > > > > > >> > > > > > > > > >> I'm not sure where you are assuming that "endpoints" only > > > refers > > > > > to > > > > > > > > >> Iceberg > > > > > > > > >> endpoints in the IRC spec. Do you have any thing you can > > point > > > > us > > > > > to > > > > > > > > >> regarding this? > > > > > > > > >> > > > > > > > > >> Best, > > > > > > > > >> Adnan Hemani > > > > > > > > >> > > > > > > > > >> On Mon, Jun 22, 2026 at 12:18 PM Dmitri Bourlatchkov < > > > > > > > [email protected]> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > >> > Hi Adam, > > > > > > > > >> > > > > > > > > > >> > > #1 - Is GET ' > > > > > > > > >> > > > > http://localhost:8181/api/catalog/v1/config?warehouse=polaris > > > > ' > > > > > > only > > > > > > > > >> > an Iceberg Catalog concept? > > > > > > > > >> > > > > > > > > > >> > My interpretation is YES, because the payload of that > > > response > > > > > is > > > > > > > > >> defined > > > > > > > > >> > by the IRC API spec [1]. > > > > > > > > >> > > > > > > > > > >> > I believe one can reasonably assume that the statement > > > > > "endpoints > > > > > > > > >> supported > > > > > > > > >> > by the server" is scoped only to the IRC API itself. > > There's > > > > no > > > > > > > > >> provision > > > > > > > > >> > in that spec about covering all possible APIs. > > > > > > > > >> > > > > > > > > > >> > That said, I'd be happy to discuss how clients can > > discover > > > > > > Polaris > > > > > > > > >> > features in general without overloading existing > > > > specifications. > > > > > > > > >> > > > > > > > > > >> > [1] > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/open-api/rest-catalog-open-api.yaml#L105 > > > > > > > > >> > > > > > > > > > >> > On Mon, Jun 22, 2026 at 2:52 PM Adam Christian < > > > > > > > > >> > [email protected]> wrote: > > > > > > > > >> > > > > > > > > > >> > > I agree with both Yufei & Dmitri: > > > > > > > > >> > > > > > > > > > > >> > > #1 - It seems like improper coupling for an Iceberg > REST > > > > > Catalog > > > > > > > > >> config > > > > > > > > >> > to > > > > > > > > >> > > return a configuration unrelated to Iceberg. > > > > > > > > >> > > #2 - Polaris is a superset of an Iceberg REST Catalog. > > > > > > > > >> > > > > > > > > > > >> > > So, in my opinion, the question is more aptly framed > as > > > two > > > > > > > > questions: > > > > > > > > >> > > > > > > > > > > >> > > #1 - Is GET ' > > > > > > > > >> > > > > http://localhost:8181/api/catalog/v1/config?warehouse=polaris > > > > > > > > >> > > ' > > > > > > > > >> > > only an Iceberg Catalog concept? > > > > > > > > >> > > #2 - Should GET ' > > > > > > > > >> > > > > > > http://localhost:8181/api/catalog/v1/config?warehouse=polaris > > > > > ' > > > > > > > > return > > > > > > > > >> > > additional endpoints? > > > > > > > > >> > > > > > > > > > > >> > > I believe that the answer to #1 is no, but that > requires > > > > > changes > > > > > > > to > > > > > > > > >> the > > > > > > > > >> > > codebase. I believe the answer to #2 is yes. > > > > > > > > >> > > > > > > > > > > >> > > Based on the codebase, we are exposing several > endpoints > > > in > > > > > this > > > > > > > > >> > > configuration API which are not Iceberg endpoints. For > > > > > example, > > > > > > in > > > > > > > > the > > > > > > > > >> > > Generic Tables case, it is explicitly not Iceberg. In > my > > > > > > opinion, > > > > > > > > >> this is > > > > > > > > >> > > alright because our API specification only says > > > > "Configuration > > > > > > > API," > > > > > > > > >> not > > > > > > > > >> > > "Iceberg Configuration API." Yes, it is > > > Iceberg-compatible, > > > > > but > > > > > > it > > > > > > > > is > > > > > > > > >> not > > > > > > > > >> > > Iceberg-centric. This retrieves the configuration for > > the > > > > > > Catalog; > > > > > > > > >> > > regardless of whether it is an Iceberg Catalog. Now, > if > > > this > > > > > is > > > > > > > > truly > > > > > > > > >> a > > > > > > > > >> > > superset, it implies that IcebergCatalogHandler.java > > > should > > > > > not > > > > > > > > handle > > > > > > > > >> > > returning the configuration. Additionally, we need to > > > > return a > > > > > > > > >> superset > > > > > > > > >> > of > > > > > > > > >> > > the ConfigResponse (an Iceberg-core concept) for the > > REST > > > > API > > > > > > > > >> response. > > > > > > > > >> > > > > > > > > > > >> > > Now, if the answer to #1 is yes, that means we should > > > > probably > > > > > > > have > > > > > > > > a > > > > > > > > >> > > separate API for configuration. > > > > > > > > >> > > > > > > > > > > >> > > What do y'all think? > > > > > > > > >> > > > > > > > > > > >> > > Go community, > > > > > > > > >> > > > > > > > > > > >> > > Adam > > > > > > > > >> > > > > > > > > > > >> > > On Mon, Jun 22, 2026 at 1:55 PM Yufei Gu < > > > > > [email protected]> > > > > > > > > >> wrote: > > > > > > > > >> > > > > > > > > > > >> > > > I think a Polaris client is also an IRC client, just > > > with > > > > > > > > additional > > > > > > > > >> > > > capabilities. In that sense, Polaris can be viewed > as > > a > > > > > > superset > > > > > > > > of > > > > > > > > >> > IRC. > > > > > > > > >> > > If > > > > > > > > >> > > > the config endpoint is intended for capability > > > discovery, > > > > > > > > returning > > > > > > > > >> > > Polaris > > > > > > > > >> > > > specific endpoints seems reasonable. Standard IRC > > > clients > > > > > can > > > > > > > > simply > > > > > > > > >> > > ignore > > > > > > > > >> > > > endpoints they don't recognize. > > > > > > > > >> > > > > > > > > > > > >> > > > A concrete example is the Polaris Spark client, > which > > > > checks > > > > > > > > server > > > > > > > > >> > > > capabilities before using Polaris specific > > > functionality: > > > > > > > > >> > > > > > > > > > > > >> > > > public List<TableIdentifier> > > listGenericTables(Namespace > > > > > ns) { > > > > > > > > >> > > > Endpoint.check(endpoints, > > > > > > > > >> PolarisEndpoints.V1_LIST_GENERIC_TABLES); > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > Yufei > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > On Fri, Jun 19, 2026 at 7:43 AM Dmitri Bourlatchkov > < > > > > > > > > >> [email protected]> > > > > > > > > >> > > > wrote: > > > > > > > > >> > > > > > > > > > > > >> > > > > Hi All, > > > > > > > > >> > > > > > > > > > > > > >> > > > > During my review of [4816] I realized [1] that > > Polaris > > > > > > returns > > > > > > > > >> some > > > > > > > > >> > > > non-IRC > > > > > > > > >> > > > > endpoints in the IRC config responses. > > > > > > > > >> > > > > > > > > > > > > >> > > > > For example: > > > > > > > > >> > > > > > > > > > > > > >> > > > > GET ' > > > > > > > > >> > > http://localhost:8181/api/catalog/v1/config?warehouse=polaris > > > ' > > > > > > > > >> > > > > > > > > > > > > >> > > > > Result: > > > > > > > > >> > > > > [...] > > > > > > > > >> > > > > "endpoints": [ > > > > > > > > >> > > > > "GET /v1/{prefix}/namespaces", > > > > > > > > >> > > > > "GET /v1/{prefix}/namespaces/{namespace}", > > > > > > > > >> > > > > "HEAD /v1/{prefix}/namespaces/{namespace}", > > > > > > > > >> > > > > [...] > > > > > > > > >> > > > > "DELETE > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > polaris/v1/{prefix}/namespaces/{namespace}/generic-tables/{generic-table}", > > > > > > > > >> > > > > "GET > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > polaris/v1/{prefix}/namespaces/{namespace}/generic-tables/{generic-table}", > > > > > > > > >> > > > > "GET > > > > > > > /polaris/v1/{prefix}/namespaces/{namespace}/policies", > > > > > > > > >> > > > > "POST > > > > > > > /polaris/v1/{prefix}/namespaces/{namespace}/policies", > > > > > > > > >> > > > > > > > > > > > > >> > > > > The latter group of endpoints is not related to > the > > > > > Iceberg > > > > > > > REST > > > > > > > > >> > > Catalog > > > > > > > > >> > > > > API. > > > > > > > > >> > > > > > > > > > > > > >> > > > > I wonder what the rationale might be for returning > > > them > > > > in > > > > > > the > > > > > > > > IRC > > > > > > > > >> > > config > > > > > > > > >> > > > > response. > > > > > > > > >> > > > > > > > > > > > > >> > > > > From my POV, returning them in the IRC config > > response > > > > is > > > > > > > > >> incorrect > > > > > > > > >> > > > because > > > > > > > > >> > > > > these endpoints form APIs that follow a different > > > > > > > specification > > > > > > > > >> and > > > > > > > > >> > > > clients > > > > > > > > >> > > > > using the IRC API do not need that information to > > > > properly > > > > > > use > > > > > > > > the > > > > > > > > >> > IRC > > > > > > > > >> > > > API. > > > > > > > > >> > > > > > > > > > > > > >> > > > > WDYT? > > > > > > > > >> > > > > > > > > > > > > >> > > > > [1] > > > > > > > > >> > > > > > > > > https://github.com/apache/polaris/pull/4816#discussion_r3438945230 > > > > > > > > >> > > > > > > > > > > > > >> > > > > [4816] > https://github.com/apache/polaris/pull/4816 > > > > > > > > >> > > > > > > > > > > > > >> > > > > Thanks, > > > > > > > > >> > > > > Dmitri. > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Dmitri Bourlatchkov > Senior Staff Software Engineer, Dremio > Dremio.com > < > https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature > > > / > Follow Us on LinkedIn <https://www.linkedin.com/company/dremio> / Get > Started <https://www.dremio.com/get-started/> > > > The Agentic Lakehouse > The only lakehouse built for agents, managed by agents >
