This is definitely simplified, but I'm providing this example to see your thinking, as I don't fully understand why you are advocating for this.
A new Realm-Level parameter is introduced that toggles whether a particular feature (and therefore its endpoints) appears in the getConfig response. For example, `ENABLE_OPENLINEAGE_ENDPOINTS` is the new feature flag. If it is on, we add the OpenLineage endpoints to the "default list" in getConfig (the param value(s) are checked during getConfig). The endpoints' path values are static already today, I believe. What is the concern with something like this? Best, Adnan Hemani On Mon, Jun 22, 2026 at 6:23 PM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Adnan, > > How will the code that produces the config payload know which API endpoints > are enabled?.. and what their paths are? > > If it just knows the full list, that creates a high level of coupling, > which I think is not sound from a modular design POV. This will complicate > code maintenance and evolution, IMHO. > > Thanks, > Dmitri. > > On Mon, Jun 22, 2026 at 8:28 PM Adnan Hemani via dev < > [email protected]> > wrote: > > > Ah yes, my mistake. But I think a similar point exists: Does the service > > need to know about all new endpoint information? Can we not make it dumb > > and gate on a server-wide feature flag or something similar? > > > > -Adnan > > > > On Mon, Jun 22, 2026 at 5:02 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi Adnan, > > > > > > You might have misinterpreted my message. My point was about the server > > > (service) side, not client side. > > > > > > Cheers, > > > Dmitri. > > > > > > On Mon, Jun 22, 2026 at 7:58 PM Adnan Hemani via dev < > > > [email protected]> > > > wrote: > > > > > > > > For example, if we are to expose Open Lineage or OSI endpoints in > IRC > > > > config, the IRC service code will have to be aware of OL / OSI > modules. > > > > > > > > I don't agree with this statement; the IRC client is always free to > > > > disregard any of these endpoints/modules. Returning this information > > does > > > > not force a client to take any action based on this extra endpoint > > > > information. > > > > > > > > I do agree that if this forced the IRC client to be fundamentally > > altered > > > > because of these additional endpoints, then that would be a > significant > > > > issue. But as Russell stated above, "what would actually break if > > someone > > > > interpreted the spec this way?" > > > > > > > > Best, > > > > Adnan Hemani > > > > > > > > On Mon, Jun 22, 2026 at 3:30 PM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > Hi Russell, > > > > > > > > > > Your points sound reasonable to me. > > > > > > > > > > Yet, noone answered my second question from my initial email. That > > is: > > > > why > > > > > do we need to expose non-IRC endpoints in the IRC config response > in > > > the > > > > > first place? > > > > > > > > > > Regarding the maintainability aspect, I see this becoming a more > > > > > substantial concern if it evolves into a self-sustaining pattern. > > > > > > > > > > For example, if we are to expose Open Lineage or OSI endpoints in > IRC > > > > > config, the IRC service code will have to be aware of OL / OSI > > > modules. I > > > > > think this dependency is conceptually incorrect. > > > > > > > > > > I do not so much mind existing non-IRC endpoints in the IRC config, > > but > > > > > extending this to new APIs feels like an anti-pattern to me. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Mon, Jun 22, 2026 at 5:46 PM Russell Spitzer < > > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > From a very pragmatic point of view, I'm not sure what we'd gain > > from > > > > > > separate > > > > > > configuration mechanisms. We already have one and it's worked > well > > so > > > > > far, > > > > > > do we have a compelling reason to swap right now? > > > > > > > > > > > > As best I can tell, the worst case is that the IRC spec later > > > tightens > > > > > the > > > > > > semantics > > > > > > of the endpoints field, making our extra entries non-compliant. > In > > > that > > > > > > situation > > > > > > we'd have to change our response, add a Polaris-specific > > > > config/discovery > > > > > > endpoint, and > > > > > > have our clients move over to it. But that's exactly the work > this > > > > > proposal > > > > > > is asking us to do today > > > > > > so we would be taking same burden now instead of potentially > never. > > > I'd > > > > > > rather defer > > > > > > it until something concrete forces our hand. > > > > > > > > > > > > The one counter I can see is that decoupling gets harder as more > > > > clients > > > > > > adopt it. But > > > > > > since the Polaris endpoints are already namespaced > (/polaris/...), > > > > > lifting > > > > > > them into a separate endpoint > > > > > > later stays mechanical, so I don't think we're meaningfully > > cheaper > > > > > doing > > > > > > it now. > > > > > > > > > > > > > > > > > > I'd like to gently point out that it doesn't feel especially > > > important > > > > > > whether the exact wording > > > > > > or original intent of the spec supports what we're doing. The > > > > discussion > > > > > is > > > > > > starting to feel > > > > > > like a legalistic reading of the spec rather than a focus on > > concrete > > > > > > impact. I'd suggest we focus > > > > > > on arguments like "what would actually break if someone > interpreted > > > the > > > > > > spec this way?" > > > > > > rather than "what was the original intent of the text?" > > > > > > > > > > > > On Mon, Jun 22, 2026 at 3:23 PM Adnan Hemani via dev < > > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > > > I think you and I view the same statement very differently - I > > > > > definitely > > > > > > > don't think it's "pretty clear that the scope of the data > > returned > > > > > under > > > > > > > that type is limited to the API defined by the spec". I see it > > this > > > > > way: > > > > > > if > > > > > > > this statement truly meant ONLY IRC endpoints, the spec would > > > > > explicitly > > > > > > > use the words "IRC spec endpoints". The absence of > clarification > > in > > > > any > > > > > > > specification rarely means you should "read-between-the-lines" > to > > > > > assume > > > > > > > the writer's "intent". > > > > > > > > > > > > > > I will look into Iceberg mailing list threads and PRs for > > > background > > > > > > > information to clarify things. > > > > > > > > > > > > > > Best, > > > > > > > Adnan Hemani > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 22, 2026 at 12:27 PM Dmitri Bourlatchkov < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > HI Adnan, > > > > > > > > > > > > > > > > The IRC spec defines a "type" for the config response. A > > property > > > > of > > > > > > that > > > > > > > > "type" is a list of "endpoints supported by the server". > There > > is > > > > no > > > > > > > > statement about generalizing those endpoints to non-IRC APIs. > > > > > > > > > > > > > > > > Therefore, I think it is pretty clear that the scope of the > > data > > > > > > returned > > > > > > > > under that type is limited to the API defined by the spec, > > which > > > is > > > > > > IRC. > > > > > > > > > > > > > > > > Let's expand this discussion a bit. > > > > > > > > > > > > > > > > What is the rationale for returning non-IRC endpoints in the > > IRC > > > > > config > > > > > > > > response? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 22, 2026 at 3:21 PM Adnan Hemani via dev < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > >> Hi Dmitri, > > > > > > > >> > > > > > > > >> I'm not sure where you are assuming that "endpoints" only > > refers > > > > to > > > > > > > >> Iceberg > > > > > > > >> endpoints in the IRC spec. Do you have any thing you can > point > > > us > > > > to > > > > > > > >> regarding this? > > > > > > > >> > > > > > > > >> Best, > > > > > > > >> Adnan Hemani > > > > > > > >> > > > > > > > >> On Mon, Jun 22, 2026 at 12:18 PM Dmitri Bourlatchkov < > > > > > > [email protected]> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> > Hi Adam, > > > > > > > >> > > > > > > > > >> > > #1 - Is GET ' > > > > > > > >> > > > http://localhost:8181/api/catalog/v1/config?warehouse=polaris > > > ' > > > > > only > > > > > > > >> > an Iceberg Catalog concept? > > > > > > > >> > > > > > > > > >> > My interpretation is YES, because the payload of that > > response > > > > is > > > > > > > >> defined > > > > > > > >> > by the IRC API spec [1]. > > > > > > > >> > > > > > > > > >> > I believe one can reasonably assume that the statement > > > > "endpoints > > > > > > > >> supported > > > > > > > >> > by the server" is scoped only to the IRC API itself. > There's > > > no > > > > > > > >> provision > > > > > > > >> > in that spec about covering all possible APIs. > > > > > > > >> > > > > > > > > >> > That said, I'd be happy to discuss how clients can > discover > > > > > Polaris > > > > > > > >> > features in general without overloading existing > > > specifications. > > > > > > > >> > > > > > > > > >> > [1] > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/open-api/rest-catalog-open-api.yaml#L105 > > > > > > > >> > > > > > > > > >> > On Mon, Jun 22, 2026 at 2:52 PM Adam Christian < > > > > > > > >> > [email protected]> wrote: > > > > > > > >> > > > > > > > > >> > > I agree with both Yufei & Dmitri: > > > > > > > >> > > > > > > > > > >> > > #1 - It seems like improper coupling for an Iceberg REST > > > > Catalog > > > > > > > >> config > > > > > > > >> > to > > > > > > > >> > > return a configuration unrelated to Iceberg. > > > > > > > >> > > #2 - Polaris is a superset of an Iceberg REST Catalog. > > > > > > > >> > > > > > > > > > >> > > So, in my opinion, the question is more aptly framed as > > two > > > > > > > questions: > > > > > > > >> > > > > > > > > > >> > > #1 - Is GET ' > > > > > > > >> > > > http://localhost:8181/api/catalog/v1/config?warehouse=polaris > > > > > > > >> > > ' > > > > > > > >> > > only an Iceberg Catalog concept? > > > > > > > >> > > #2 - Should GET ' > > > > > > > >> > > > > > http://localhost:8181/api/catalog/v1/config?warehouse=polaris > > > > ' > > > > > > > return > > > > > > > >> > > additional endpoints? > > > > > > > >> > > > > > > > > > >> > > I believe that the answer to #1 is no, but that requires > > > > changes > > > > > > to > > > > > > > >> the > > > > > > > >> > > codebase. I believe the answer to #2 is yes. > > > > > > > >> > > > > > > > > > >> > > Based on the codebase, we are exposing several endpoints > > in > > > > this > > > > > > > >> > > configuration API which are not Iceberg endpoints. For > > > > example, > > > > > in > > > > > > > the > > > > > > > >> > > Generic Tables case, it is explicitly not Iceberg. In my > > > > > opinion, > > > > > > > >> this is > > > > > > > >> > > alright because our API specification only says > > > "Configuration > > > > > > API," > > > > > > > >> not > > > > > > > >> > > "Iceberg Configuration API." Yes, it is > > Iceberg-compatible, > > > > but > > > > > it > > > > > > > is > > > > > > > >> not > > > > > > > >> > > Iceberg-centric. This retrieves the configuration for > the > > > > > Catalog; > > > > > > > >> > > regardless of whether it is an Iceberg Catalog. Now, if > > this > > > > is > > > > > > > truly > > > > > > > >> a > > > > > > > >> > > superset, it implies that IcebergCatalogHandler.java > > should > > > > not > > > > > > > handle > > > > > > > >> > > returning the configuration. Additionally, we need to > > > return a > > > > > > > >> superset > > > > > > > >> > of > > > > > > > >> > > the ConfigResponse (an Iceberg-core concept) for the > REST > > > API > > > > > > > >> response. > > > > > > > >> > > > > > > > > > >> > > Now, if the answer to #1 is yes, that means we should > > > probably > > > > > > have > > > > > > > a > > > > > > > >> > > separate API for configuration. > > > > > > > >> > > > > > > > > > >> > > What do y'all think? > > > > > > > >> > > > > > > > > > >> > > Go community, > > > > > > > >> > > > > > > > > > >> > > Adam > > > > > > > >> > > > > > > > > > >> > > On Mon, Jun 22, 2026 at 1:55 PM Yufei Gu < > > > > [email protected]> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > > >> > > > I think a Polaris client is also an IRC client, just > > with > > > > > > > additional > > > > > > > >> > > > capabilities. In that sense, Polaris can be viewed as > a > > > > > superset > > > > > > > of > > > > > > > >> > IRC. > > > > > > > >> > > If > > > > > > > >> > > > the config endpoint is intended for capability > > discovery, > > > > > > > returning > > > > > > > >> > > Polaris > > > > > > > >> > > > specific endpoints seems reasonable. Standard IRC > > clients > > > > can > > > > > > > simply > > > > > > > >> > > ignore > > > > > > > >> > > > endpoints they don't recognize. > > > > > > > >> > > > > > > > > > > >> > > > A concrete example is the Polaris Spark client, which > > > checks > > > > > > > server > > > > > > > >> > > > capabilities before using Polaris specific > > functionality: > > > > > > > >> > > > > > > > > > > >> > > > public List<TableIdentifier> > listGenericTables(Namespace > > > > ns) { > > > > > > > >> > > > Endpoint.check(endpoints, > > > > > > > >> PolarisEndpoints.V1_LIST_GENERIC_TABLES); > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > Yufei > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > On Fri, Jun 19, 2026 at 7:43 AM Dmitri Bourlatchkov < > > > > > > > >> [email protected]> > > > > > > > >> > > > wrote: > > > > > > > >> > > > > > > > > > > >> > > > > Hi All, > > > > > > > >> > > > > > > > > > > > >> > > > > During my review of [4816] I realized [1] that > Polaris > > > > > returns > > > > > > > >> some > > > > > > > >> > > > non-IRC > > > > > > > >> > > > > endpoints in the IRC config responses. > > > > > > > >> > > > > > > > > > > > >> > > > > For example: > > > > > > > >> > > > > > > > > > > > >> > > > > GET ' > > > > > > > >> > http://localhost:8181/api/catalog/v1/config?warehouse=polaris > > ' > > > > > > > >> > > > > > > > > > > > >> > > > > Result: > > > > > > > >> > > > > [...] > > > > > > > >> > > > > "endpoints": [ > > > > > > > >> > > > > "GET /v1/{prefix}/namespaces", > > > > > > > >> > > > > "GET /v1/{prefix}/namespaces/{namespace}", > > > > > > > >> > > > > "HEAD /v1/{prefix}/namespaces/{namespace}", > > > > > > > >> > > > > [...] > > > > > > > >> > > > > "DELETE > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > polaris/v1/{prefix}/namespaces/{namespace}/generic-tables/{generic-table}", > > > > > > > >> > > > > "GET > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > polaris/v1/{prefix}/namespaces/{namespace}/generic-tables/{generic-table}", > > > > > > > >> > > > > "GET > > > > > > /polaris/v1/{prefix}/namespaces/{namespace}/policies", > > > > > > > >> > > > > "POST > > > > > > /polaris/v1/{prefix}/namespaces/{namespace}/policies", > > > > > > > >> > > > > > > > > > > > >> > > > > The latter group of endpoints is not related to the > > > > Iceberg > > > > > > REST > > > > > > > >> > > Catalog > > > > > > > >> > > > > API. > > > > > > > >> > > > > > > > > > > > >> > > > > I wonder what the rationale might be for returning > > them > > > in > > > > > the > > > > > > > IRC > > > > > > > >> > > config > > > > > > > >> > > > > response. > > > > > > > >> > > > > > > > > > > > >> > > > > From my POV, returning them in the IRC config > response > > > is > > > > > > > >> incorrect > > > > > > > >> > > > because > > > > > > > >> > > > > these endpoints form APIs that follow a different > > > > > > specification > > > > > > > >> and > > > > > > > >> > > > clients > > > > > > > >> > > > > using the IRC API do not need that information to > > > properly > > > > > use > > > > > > > the > > > > > > > >> > IRC > > > > > > > >> > > > API. > > > > > > > >> > > > > > > > > > > > >> > > > > WDYT? > > > > > > > >> > > > > > > > > > > > >> > > > > [1] > > > > > > > >> > > > > > > https://github.com/apache/polaris/pull/4816#discussion_r3438945230 > > > > > > > >> > > > > > > > > > > > >> > > > > [4816] https://github.com/apache/polaris/pull/4816 > > > > > > > >> > > > > > > > > > > > >> > > > > Thanks, > > > > > > > >> > > > > Dmitri. > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
