Hi everyone, Thanks for the proposal and all the great discussion! I also agree that we should separate Polaris-only APIs from the IRC spec to keep the IRC spec consistent with the upstream version. I’d like to propose another way to achieve this, as some APIs may not need a separate prefix but are still better kept in separate YAML files.
This is particularly useful for cases where new APIs logically belong under `/catalog` (e.g., the notification API) but should remain separate from the IRC YAML file for practical reasons. Similar cases will likely arise in the future. To address this, we can split the OpenAPI spec into multiple YAML files while keeping them grouped under the same prefix. This allows Polaris-specific APIs to stay separate while ensuring the Iceberg spec remains untouched. I tested this approach using policy management APIs in PR #856 <https://github.com/apache/polaris/pull/856>[1], based on this blog post <https://techdocs.studio/blog/how-to-split-open-api-spec-into-multiple-files> [2]. With this approach, when a Polaris catalog-level API is later contributed to Iceberg, we don’t need to re-route anything. We simply pull the latest version from IRC (assuming it is merged) and remove it from the Polaris YAML file. The API definitions remain the same—only their location changes. The downside of this approach is that it introduces additional references across multiple YAML files. Would love to hear your thoughts on this! Best regards, Honah [1] https://github.com/apache/polaris/pull/856 [2] https://techdocs.studio/blog/how-to-split-open-api-spec-into-multiple-files On Thu, Jan 23, 2025 at 2:22 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Jack, > > Welcome back! > > I like your idea to separate Iceberg REST Apis and Polaris REST Apis. > It's cleaner and it will certainly simplify the maintenance/update > (especially for the Iceberg REST APIs). > > About the Polaris REST APIs, that's the idea: we start first in > Polaris but there is good chance to be proposed to Iceberg REST Spec. > As the "pace" between implementation and spec is different, the > purpose here is to start with an implementation and propose a generic > spec later. > > I think we should also cleanly state the "scope" of the APIs related > to prefix. Imho, we are gathering too much APIs in /catalog and > /management: I think we should consider more fine-grained prefix > and/or inner prefix, else we may have to create a bunch of routes from > one prefix to another (when an API move from Polaris to Iceberg etc). > > Regards > JB > > On Wed, Jan 22, 2025 at 10:39 PM Yufei Gu <flyrain...@gmail.com> wrote: > > > > Thanks, Jack, for bringing this up! I think separating the Iceberg REST > > APIs from the other Polaris REST APIs is an excellent idea. Not only does > > it simplify any potential future rebases of the Iceberg REST spec, but it > > also provides developers with a clearer distinction between the different > > sets of REST APIs. > > > > To implement this, we may need to experiment with the auto-generation > > tools. Feel free to file issues, or propose ideas. > > > > Yufei > > > > > > On Wed, Jan 22, 2025 at 1:30 PM Jack Ye <yezhao...@gmail.com> wrote: > > > > > Hi everyone, > > > > > > I am starting to work on Polaris related tasks, and realize that > currently > > > the state of the spec folder [1] in the project is a bit disorganized: > > > > > > The Iceberg REST catalog (IRC) spec [2] contains non-standard APIs > like the > > > /notifications API, but misses a few APIs like the scan planning ones. > > > > > > There is the Polaris management service API spec [3] that seems to > record > > > the identity management APIs in Polaris and will never be a part of > the IRC > > > spec. But I see new proposals like policy [4] and volume [5] that will > > > likely introduce new APIs, and I am not sure where those APIs would go, > > > will they be in separate OpenAPI YAML files, or a part of the > management > > > service APIs, or do we expect them to be eventually a part of the > official > > > IRC spec? > > > > > > I propose that we should agree upon a consistent framework to deal with > > > these APIs, to allow users to more clearly understand different types > of > > > APIs in Polaris, and allow developers to more easily and more > consistently > > > add new ones or update existing ones. The framework I have in mind is > as > > > the following: > > > > > > 1. there will be in general 2 types of APIs: IRC APIs and Polaris > APIs. The > > > IRC APIs are precisely following the IRC spec, whereas the Polaris > APIs are > > > only available in a Polaris server. It is possible for some Polaris > APIs to > > > be developed first to unblock support of certain use cases, and later > > > contributed back to the IRC spec, depending on the appetite of both > > > communities. If that happens, the Polaris API will be deprecated in > favor > > > of the official IRC API when it gets accepted in Iceberg. > > > > > > 2. For Polaris APIs, it is subdivided into different groups by > > > functionality, and these groups are physically separated into different > > > YAML files. Currently there are 2 groups: > > > - Polaris management service > > > - Polaris notification service > > > > > > And there can be new groups in the future for policy or volume if those > > > proposals get accepted in Polaris. > > > > > > What do we think? > > > > > > Best, > > > Jack Ye > > > > > > [1] https://github.com/apache/polaris/tree/main/spec > > > [2] > > > > https://github.com/apache/polaris/blob/main/spec/rest-catalog-open-api.yaml > > > [3] > > > > > > > https://github.com/apache/polaris/blob/main/spec/polaris-management-service.yml > > > [4] > > > > > > > https://docs.google.com/document/d/1kIiVkFFg9tPa5SH70b9WwzbmclrzH3qWHKfCKXw5lbs/edit?tab=t.0#heading=h.nly223xz13km > > > [5] > > > > > > > https://docs.google.com/document/d/1ofljkrtiXRWc-v6hfkg_laKlYltepTPX7zsg44Tb-BY/edit?tab=t.0#heading=h.7ic5c343eju1 > > > >
