> It is possible for some Polaris APIs to be developed first to unblock support of certain use cases, and later contributed back to the IRC spec, depending on the appetite of both communities.
This was exactly the intention with the notification API. It was never expected to live long-term in the Polaris spec. The folks who worked on that haven’t yet gone through the work to try to contribute it upstream. The expectation is exactly as you described. What you call the “Polaris APIs” are what we’ve been calling the “management APIs”. The catalog API spec is intended to be a replica of the upstream Iceberg catalog spec - albeit a bit out of date (the scan plan API hadn’t been adopted when we created the project). The only real out of place API is the notification API. I would be fine to move that out into its own spec file until the upstream contribution process is complete. That makes it easier to update the spec, as Yufei mentioned. The new features you mentioned do introduce some challenges to the current organization. Today, we use /management and /catalog to separate the two sets of APIs. That makes less sense when we need to modify child entities under parents that are defined by the Iceberg catalog spec (e.g., attach a policy to a table or get the volumes from a catalog). For those, I think separate spec files makes a lot of sense. So management spec captures APIs that deal with principals, roles, and grants, Iceberg spec reflects upstream REST catalog spec, and notifications has the extra notifications API. There is still a question of which api prefix is used by these APIs. E.g., when adding a policy to a table, I can see adding something under /catalog, but what about creating a new policy? Is that a /management prefix? Or a new prefix altogether? I do like the separation between the Polaris APIs and the Iceberg ones, so I’d be reluctant to put things under the /catalog prefix. I’d propose we continue to use /management to separate entities that are native to Polaris and only use /catalog when dealing with Iceberg entities. Thoughts? Mike On Wed, Jan 22, 2025 at 1:40 PM Yufei Gu <flyrain...@gmail.com> wrote: > Thanks, Jack, for bringing this up! I think separating the Iceberg REST > APIs from the other Polaris REST APIs is an excellent idea. Not only does > it simplify any potential future rebases of the Iceberg REST spec, but it > also provides developers with a clearer distinction between the different > sets of REST APIs. > > To implement this, we may need to experiment with the auto-generation > tools. Feel free to file issues, or propose ideas. > > Yufei > > > On Wed, Jan 22, 2025 at 1:30 PM Jack Ye <yezhao...@gmail.com> wrote: > > > Hi everyone, > > > > I am starting to work on Polaris related tasks, and realize that > currently > > the state of the spec folder [1] in the project is a bit disorganized: > > > > The Iceberg REST catalog (IRC) spec [2] contains non-standard APIs like > the > > /notifications API, but misses a few APIs like the scan planning ones. > > > > There is the Polaris management service API spec [3] that seems to record > > the identity management APIs in Polaris and will never be a part of the > IRC > > spec. But I see new proposals like policy [4] and volume [5] that will > > likely introduce new APIs, and I am not sure where those APIs would go, > > will they be in separate OpenAPI YAML files, or a part of the management > > service APIs, or do we expect them to be eventually a part of the > official > > IRC spec? > > > > I propose that we should agree upon a consistent framework to deal with > > these APIs, to allow users to more clearly understand different types of > > APIs in Polaris, and allow developers to more easily and more > consistently > > add new ones or update existing ones. The framework I have in mind is as > > the following: > > > > 1. there will be in general 2 types of APIs: IRC APIs and Polaris APIs. > The > > IRC APIs are precisely following the IRC spec, whereas the Polaris APIs > are > > only available in a Polaris server. It is possible for some Polaris APIs > to > > be developed first to unblock support of certain use cases, and later > > contributed back to the IRC spec, depending on the appetite of both > > communities. If that happens, the Polaris API will be deprecated in favor > > of the official IRC API when it gets accepted in Iceberg. > > > > 2. For Polaris APIs, it is subdivided into different groups by > > functionality, and these groups are physically separated into different > > YAML files. Currently there are 2 groups: > > - Polaris management service > > - Polaris notification service > > > > And there can be new groups in the future for policy or volume if those > > proposals get accepted in Polaris. > > > > What do we think? > > > > Best, > > Jack Ye > > > > [1] https://github.com/apache/polaris/tree/main/spec > > [2] > > > https://github.com/apache/polaris/blob/main/spec/rest-catalog-open-api.yaml > > [3] > > > > > https://github.com/apache/polaris/blob/main/spec/polaris-management-service.yml > > [4] > > > > > https://docs.google.com/document/d/1kIiVkFFg9tPa5SH70b9WwzbmclrzH3qWHKfCKXw5lbs/edit?tab=t.0#heading=h.nly223xz13km > > [5] > > > > > https://docs.google.com/document/d/1ofljkrtiXRWc-v6hfkg_laKlYltepTPX7zsg44Tb-BY/edit?tab=t.0#heading=h.7ic5c343eju1 > > >
