Thanks Honah for picking it up. The approach looks good to me overall. Left some comments in the PR.
Yufei On Tue, Jan 28, 2025 at 7:22 AM Dmitri Bourlatchkov <di...@apache.org> wrote: > Using multiple YAML files per API / API area sounds good to me. > > We should definitely use the Iceberg REST API YAML without any > modifications to avoid accidental mistakes. > > Cheers, > Dmitri. > > On Tue, Jan 28, 2025 at 2:28 AM Honah J. <hon...@apache.org> wrote: > > > Hi everyone, > > > > Thanks for the proposal and all the great discussion! I also agree that > we > > should separate Polaris-only APIs from the IRC spec to keep the IRC spec > > consistent with the upstream version. I’d like to propose another way to > > achieve this, as some APIs may not need a separate prefix but are still > > better kept in separate YAML files. > > > > This is particularly useful for cases where new APIs logically belong > under > > `/catalog` (e.g., the notification API) but should remain separate from > the > > IRC YAML file for practical reasons. Similar cases will likely arise in > the > > future. > > > > To address this, we can split the OpenAPI spec into multiple YAML files > > while keeping them grouped under the same prefix. This allows > > Polaris-specific APIs to stay separate while ensuring the Iceberg spec > > remains untouched. I tested this approach using policy management APIs in > > PR > > #856 <https://github.com/apache/polaris/pull/856>[1], based on this blog > > post > > < > > > https://techdocs.studio/blog/how-to-split-open-api-spec-into-multiple-files > > > > > [2]. > > > > With this approach, when a Polaris catalog-level API is later contributed > > to Iceberg, we don’t need to re-route anything. We simply pull the latest > > version from IRC (assuming it is merged) and remove it from the Polaris > > YAML file. The API definitions remain the same—only their location > changes. > > The downside of this approach is that it introduces additional references > > across multiple YAML files. > > > > Would love to hear your thoughts on this! > > > > Best regards, > > Honah > > > > [1] https://github.com/apache/polaris/pull/856 > > [2] > > > https://techdocs.studio/blog/how-to-split-open-api-spec-into-multiple-files > > > > > > On Thu, Jan 23, 2025 at 2:22 AM Jean-Baptiste Onofré <j...@nanthrax.net> > > wrote: > > > > > Hi Jack, > > > > > > Welcome back! > > > > > > I like your idea to separate Iceberg REST Apis and Polaris REST Apis. > > > It's cleaner and it will certainly simplify the maintenance/update > > > (especially for the Iceberg REST APIs). > > > > > > About the Polaris REST APIs, that's the idea: we start first in > > > Polaris but there is good chance to be proposed to Iceberg REST Spec. > > > As the "pace" between implementation and spec is different, the > > > purpose here is to start with an implementation and propose a generic > > > spec later. > > > > > > I think we should also cleanly state the "scope" of the APIs related > > > to prefix. Imho, we are gathering too much APIs in /catalog and > > > /management: I think we should consider more fine-grained prefix > > > and/or inner prefix, else we may have to create a bunch of routes from > > > one prefix to another (when an API move from Polaris to Iceberg etc). > > > > > > Regards > > > JB > > > > > > On Wed, Jan 22, 2025 at 10:39 PM Yufei Gu <flyrain...@gmail.com> > wrote: > > > > > > > > Thanks, Jack, for bringing this up! I think separating the Iceberg > REST > > > > APIs from the other Polaris REST APIs is an excellent idea. Not only > > does > > > > it simplify any potential future rebases of the Iceberg REST spec, > but > > it > > > > also provides developers with a clearer distinction between the > > different > > > > sets of REST APIs. > > > > > > > > To implement this, we may need to experiment with the auto-generation > > > > tools. Feel free to file issues, or propose ideas. > > > > > > > > Yufei > > > > > > > > > > > > On Wed, Jan 22, 2025 at 1:30 PM Jack Ye <yezhao...@gmail.com> wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > I am starting to work on Polaris related tasks, and realize that > > > currently > > > > > the state of the spec folder [1] in the project is a bit > > disorganized: > > > > > > > > > > The Iceberg REST catalog (IRC) spec [2] contains non-standard APIs > > > like the > > > > > /notifications API, but misses a few APIs like the scan planning > > ones. > > > > > > > > > > There is the Polaris management service API spec [3] that seems to > > > record > > > > > the identity management APIs in Polaris and will never be a part of > > > the IRC > > > > > spec. But I see new proposals like policy [4] and volume [5] that > > will > > > > > likely introduce new APIs, and I am not sure where those APIs would > > go, > > > > > will they be in separate OpenAPI YAML files, or a part of the > > > management > > > > > service APIs, or do we expect them to be eventually a part of the > > > official > > > > > IRC spec? > > > > > > > > > > I propose that we should agree upon a consistent framework to deal > > with > > > > > these APIs, to allow users to more clearly understand different > types > > > of > > > > > APIs in Polaris, and allow developers to more easily and more > > > consistently > > > > > add new ones or update existing ones. The framework I have in mind > is > > > as > > > > > the following: > > > > > > > > > > 1. there will be in general 2 types of APIs: IRC APIs and Polaris > > > APIs. The > > > > > IRC APIs are precisely following the IRC spec, whereas the Polaris > > > APIs are > > > > > only available in a Polaris server. It is possible for some Polaris > > > APIs to > > > > > be developed first to unblock support of certain use cases, and > later > > > > > contributed back to the IRC spec, depending on the appetite of both > > > > > communities. If that happens, the Polaris API will be deprecated in > > > favor > > > > > of the official IRC API when it gets accepted in Iceberg. > > > > > > > > > > 2. For Polaris APIs, it is subdivided into different groups by > > > > > functionality, and these groups are physically separated into > > different > > > > > YAML files. Currently there are 2 groups: > > > > > - Polaris management service > > > > > - Polaris notification service > > > > > > > > > > And there can be new groups in the future for policy or volume if > > those > > > > > proposals get accepted in Polaris. > > > > > > > > > > What do we think? > > > > > > > > > > Best, > > > > > Jack Ye > > > > > > > > > > [1] https://github.com/apache/polaris/tree/main/spec > > > > > [2] > > > > > > > > > > > https://github.com/apache/polaris/blob/main/spec/rest-catalog-open-api.yaml > > > > > [3] > > > > > > > > > > > > > > > > https://github.com/apache/polaris/blob/main/spec/polaris-management-service.yml > > > > > [4] > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1kIiVkFFg9tPa5SH70b9WwzbmclrzH3qWHKfCKXw5lbs/edit?tab=t.0#heading=h.nly223xz13km > > > > > [5] > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1ofljkrtiXRWc-v6hfkg_laKlYltepTPX7zsg44Tb-BY/edit?tab=t.0#heading=h.7ic5c343eju1 > > > > > > > > > > >
