Hi Jack, Welcome back!
I like your idea to separate Iceberg REST Apis and Polaris REST Apis. It's cleaner and it will certainly simplify the maintenance/update (especially for the Iceberg REST APIs). About the Polaris REST APIs, that's the idea: we start first in Polaris but there is good chance to be proposed to Iceberg REST Spec. As the "pace" between implementation and spec is different, the purpose here is to start with an implementation and propose a generic spec later. I think we should also cleanly state the "scope" of the APIs related to prefix. Imho, we are gathering too much APIs in /catalog and /management: I think we should consider more fine-grained prefix and/or inner prefix, else we may have to create a bunch of routes from one prefix to another (when an API move from Polaris to Iceberg etc). Regards JB On Wed, Jan 22, 2025 at 10:39 PM Yufei Gu <flyrain...@gmail.com> wrote: > > Thanks, Jack, for bringing this up! I think separating the Iceberg REST > APIs from the other Polaris REST APIs is an excellent idea. Not only does > it simplify any potential future rebases of the Iceberg REST spec, but it > also provides developers with a clearer distinction between the different > sets of REST APIs. > > To implement this, we may need to experiment with the auto-generation > tools. Feel free to file issues, or propose ideas. > > Yufei > > > On Wed, Jan 22, 2025 at 1:30 PM Jack Ye <yezhao...@gmail.com> wrote: > > > Hi everyone, > > > > I am starting to work on Polaris related tasks, and realize that currently > > the state of the spec folder [1] in the project is a bit disorganized: > > > > The Iceberg REST catalog (IRC) spec [2] contains non-standard APIs like the > > /notifications API, but misses a few APIs like the scan planning ones. > > > > There is the Polaris management service API spec [3] that seems to record > > the identity management APIs in Polaris and will never be a part of the IRC > > spec. But I see new proposals like policy [4] and volume [5] that will > > likely introduce new APIs, and I am not sure where those APIs would go, > > will they be in separate OpenAPI YAML files, or a part of the management > > service APIs, or do we expect them to be eventually a part of the official > > IRC spec? > > > > I propose that we should agree upon a consistent framework to deal with > > these APIs, to allow users to more clearly understand different types of > > APIs in Polaris, and allow developers to more easily and more consistently > > add new ones or update existing ones. The framework I have in mind is as > > the following: > > > > 1. there will be in general 2 types of APIs: IRC APIs and Polaris APIs. The > > IRC APIs are precisely following the IRC spec, whereas the Polaris APIs are > > only available in a Polaris server. It is possible for some Polaris APIs to > > be developed first to unblock support of certain use cases, and later > > contributed back to the IRC spec, depending on the appetite of both > > communities. If that happens, the Polaris API will be deprecated in favor > > of the official IRC API when it gets accepted in Iceberg. > > > > 2. For Polaris APIs, it is subdivided into different groups by > > functionality, and these groups are physically separated into different > > YAML files. Currently there are 2 groups: > > - Polaris management service > > - Polaris notification service > > > > And there can be new groups in the future for policy or volume if those > > proposals get accepted in Polaris. > > > > What do we think? > > > > Best, > > Jack Ye > > > > [1] https://github.com/apache/polaris/tree/main/spec > > [2] > > https://github.com/apache/polaris/blob/main/spec/rest-catalog-open-api.yaml > > [3] > > > > https://github.com/apache/polaris/blob/main/spec/polaris-management-service.yml > > [4] > > > > https://docs.google.com/document/d/1kIiVkFFg9tPa5SH70b9WwzbmclrzH3qWHKfCKXw5lbs/edit?tab=t.0#heading=h.nly223xz13km > > [5] > > > > https://docs.google.com/document/d/1ofljkrtiXRWc-v6hfkg_laKlYltepTPX7zsg44Tb-BY/edit?tab=t.0#heading=h.7ic5c343eju1 > >
