Using multiple YAML files per API / API area sounds good to me. We should definitely use the Iceberg REST API YAML without any modifications to avoid accidental mistakes.
Cheers, Dmitri. On Tue, Jan 28, 2025 at 2:28 AM Honah J. <hon...@apache.org> wrote: > Hi everyone, > > Thanks for the proposal and all the great discussion! I also agree that we > should separate Polaris-only APIs from the IRC spec to keep the IRC spec > consistent with the upstream version. I’d like to propose another way to > achieve this, as some APIs may not need a separate prefix but are still > better kept in separate YAML files. > > This is particularly useful for cases where new APIs logically belong under > `/catalog` (e.g., the notification API) but should remain separate from the > IRC YAML file for practical reasons. Similar cases will likely arise in the > future. > > To address this, we can split the OpenAPI spec into multiple YAML files > while keeping them grouped under the same prefix. This allows > Polaris-specific APIs to stay separate while ensuring the Iceberg spec > remains untouched. I tested this approach using policy management APIs in > PR > #856 <https://github.com/apache/polaris/pull/856>[1], based on this blog > post > < > https://techdocs.studio/blog/how-to-split-open-api-spec-into-multiple-files > > > [2]. > > With this approach, when a Polaris catalog-level API is later contributed > to Iceberg, we don’t need to re-route anything. We simply pull the latest > version from IRC (assuming it is merged) and remove it from the Polaris > YAML file. The API definitions remain the same—only their location changes. > The downside of this approach is that it introduces additional references > across multiple YAML files. > > Would love to hear your thoughts on this! > > Best regards, > Honah > > [1] https://github.com/apache/polaris/pull/856 > [2] > https://techdocs.studio/blog/how-to-split-open-api-spec-into-multiple-files > > > On Thu, Jan 23, 2025 at 2:22 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > Hi Jack, > > > > Welcome back! > > > > I like your idea to separate Iceberg REST Apis and Polaris REST Apis. > > It's cleaner and it will certainly simplify the maintenance/update > > (especially for the Iceberg REST APIs). > > > > About the Polaris REST APIs, that's the idea: we start first in > > Polaris but there is good chance to be proposed to Iceberg REST Spec. > > As the "pace" between implementation and spec is different, the > > purpose here is to start with an implementation and propose a generic > > spec later. > > > > I think we should also cleanly state the "scope" of the APIs related > > to prefix. Imho, we are gathering too much APIs in /catalog and > > /management: I think we should consider more fine-grained prefix > > and/or inner prefix, else we may have to create a bunch of routes from > > one prefix to another (when an API move from Polaris to Iceberg etc). > > > > Regards > > JB > > > > On Wed, Jan 22, 2025 at 10:39 PM Yufei Gu <flyrain...@gmail.com> wrote: > > > > > > Thanks, Jack, for bringing this up! I think separating the Iceberg REST > > > APIs from the other Polaris REST APIs is an excellent idea. Not only > does > > > it simplify any potential future rebases of the Iceberg REST spec, but > it > > > also provides developers with a clearer distinction between the > different > > > sets of REST APIs. > > > > > > To implement this, we may need to experiment with the auto-generation > > > tools. Feel free to file issues, or propose ideas. > > > > > > Yufei > > > > > > > > > On Wed, Jan 22, 2025 at 1:30 PM Jack Ye <yezhao...@gmail.com> wrote: > > > > > > > Hi everyone, > > > > > > > > I am starting to work on Polaris related tasks, and realize that > > currently > > > > the state of the spec folder [1] in the project is a bit > disorganized: > > > > > > > > The Iceberg REST catalog (IRC) spec [2] contains non-standard APIs > > like the > > > > /notifications API, but misses a few APIs like the scan planning > ones. > > > > > > > > There is the Polaris management service API spec [3] that seems to > > record > > > > the identity management APIs in Polaris and will never be a part of > > the IRC > > > > spec. But I see new proposals like policy [4] and volume [5] that > will > > > > likely introduce new APIs, and I am not sure where those APIs would > go, > > > > will they be in separate OpenAPI YAML files, or a part of the > > management > > > > service APIs, or do we expect them to be eventually a part of the > > official > > > > IRC spec? > > > > > > > > I propose that we should agree upon a consistent framework to deal > with > > > > these APIs, to allow users to more clearly understand different types > > of > > > > APIs in Polaris, and allow developers to more easily and more > > consistently > > > > add new ones or update existing ones. The framework I have in mind is > > as > > > > the following: > > > > > > > > 1. there will be in general 2 types of APIs: IRC APIs and Polaris > > APIs. The > > > > IRC APIs are precisely following the IRC spec, whereas the Polaris > > APIs are > > > > only available in a Polaris server. It is possible for some Polaris > > APIs to > > > > be developed first to unblock support of certain use cases, and later > > > > contributed back to the IRC spec, depending on the appetite of both > > > > communities. If that happens, the Polaris API will be deprecated in > > favor > > > > of the official IRC API when it gets accepted in Iceberg. > > > > > > > > 2. For Polaris APIs, it is subdivided into different groups by > > > > functionality, and these groups are physically separated into > different > > > > YAML files. Currently there are 2 groups: > > > > - Polaris management service > > > > - Polaris notification service > > > > > > > > And there can be new groups in the future for policy or volume if > those > > > > proposals get accepted in Polaris. > > > > > > > > What do we think? > > > > > > > > Best, > > > > Jack Ye > > > > > > > > [1] https://github.com/apache/polaris/tree/main/spec > > > > [2] > > > > > > > https://github.com/apache/polaris/blob/main/spec/rest-catalog-open-api.yaml > > > > [3] > > > > > > > > > > > https://github.com/apache/polaris/blob/main/spec/polaris-management-service.yml > > > > [4] > > > > > > > > > > > https://docs.google.com/document/d/1kIiVkFFg9tPa5SH70b9WwzbmclrzH3qWHKfCKXw5lbs/edit?tab=t.0#heading=h.nly223xz13km > > > > [5] > > > > > > > > > > > https://docs.google.com/document/d/1ofljkrtiXRWc-v6hfkg_laKlYltepTPX7zsg44Tb-BY/edit?tab=t.0#heading=h.7ic5c343eju1 > > > > > > >
