Hi everyone, I am starting to work on Polaris related tasks, and realize that currently the state of the spec folder [1] in the project is a bit disorganized:
The Iceberg REST catalog (IRC) spec [2] contains non-standard APIs like the /notifications API, but misses a few APIs like the scan planning ones. There is the Polaris management service API spec [3] that seems to record the identity management APIs in Polaris and will never be a part of the IRC spec. But I see new proposals like policy [4] and volume [5] that will likely introduce new APIs, and I am not sure where those APIs would go, will they be in separate OpenAPI YAML files, or a part of the management service APIs, or do we expect them to be eventually a part of the official IRC spec? I propose that we should agree upon a consistent framework to deal with these APIs, to allow users to more clearly understand different types of APIs in Polaris, and allow developers to more easily and more consistently add new ones or update existing ones. The framework I have in mind is as the following: 1. there will be in general 2 types of APIs: IRC APIs and Polaris APIs. The IRC APIs are precisely following the IRC spec, whereas the Polaris APIs are only available in a Polaris server. It is possible for some Polaris APIs to be developed first to unblock support of certain use cases, and later contributed back to the IRC spec, depending on the appetite of both communities. If that happens, the Polaris API will be deprecated in favor of the official IRC API when it gets accepted in Iceberg. 2. For Polaris APIs, it is subdivided into different groups by functionality, and these groups are physically separated into different YAML files. Currently there are 2 groups: - Polaris management service - Polaris notification service And there can be new groups in the future for policy or volume if those proposals get accepted in Polaris. What do we think? Best, Jack Ye [1] https://github.com/apache/polaris/tree/main/spec [2] https://github.com/apache/polaris/blob/main/spec/rest-catalog-open-api.yaml [3] https://github.com/apache/polaris/blob/main/spec/polaris-management-service.yml [4] https://docs.google.com/document/d/1kIiVkFFg9tPa5SH70b9WwzbmclrzH3qWHKfCKXw5lbs/edit?tab=t.0#heading=h.nly223xz13km [5] https://docs.google.com/document/d/1ofljkrtiXRWc-v6hfkg_laKlYltepTPX7zsg44Tb-BY/edit?tab=t.0#heading=h.7ic5c343eju1
