Hi everyone, Thanks for the feedback! I've created PR: https://github.com/apache/polaris/pull/906 to implement the proposed solution. I've also linked the PR to the issue pointed out by Robert.
Best regards, Honah On Wed, Jan 29, 2025 at 1:54 AM Robert Stupp <sn...@snazy.de> wrote: > Noting: there's an issue to track this: > https://github.com/apache/polaris/issues/553 > > On 28.01.25 21:24, Yufei Gu wrote: > > Thanks Honah for picking it up. The approach looks good to me overall. > Left > > some comments in the PR. > > > > Yufei > > > > > > On Tue, Jan 28, 2025 at 7:22 AM Dmitri Bourlatchkov <di...@apache.org> > > wrote: > > > >> Using multiple YAML files per API / API area sounds good to me. > >> > >> We should definitely use the Iceberg REST API YAML without any > >> modifications to avoid accidental mistakes. > >> > >> Cheers, > >> Dmitri. > >> > >> On Tue, Jan 28, 2025 at 2:28 AM Honah J. <hon...@apache.org> wrote: > >> > >>> Hi everyone, > >>> > >>> Thanks for the proposal and all the great discussion! I also agree that > >> we > >>> should separate Polaris-only APIs from the IRC spec to keep the IRC > spec > >>> consistent with the upstream version. I’d like to propose another way > to > >>> achieve this, as some APIs may not need a separate prefix but are still > >>> better kept in separate YAML files. > >>> > >>> This is particularly useful for cases where new APIs logically belong > >> under > >>> `/catalog` (e.g., the notification API) but should remain separate from > >> the > >>> IRC YAML file for practical reasons. Similar cases will likely arise in > >> the > >>> future. > >>> > >>> To address this, we can split the OpenAPI spec into multiple YAML files > >>> while keeping them grouped under the same prefix. This allows > >>> Polaris-specific APIs to stay separate while ensuring the Iceberg spec > >>> remains untouched. I tested this approach using policy management APIs > in > >>> PR > >>> #856 <https://github.com/apache/polaris/pull/856>[1], based on this > blog > >>> post > >>> < > >>> > >> > https://techdocs.studio/blog/how-to-split-open-api-spec-into-multiple-files > >>> [2]. > >>> > >>> With this approach, when a Polaris catalog-level API is later > contributed > >>> to Iceberg, we don’t need to re-route anything. We simply pull the > latest > >>> version from IRC (assuming it is merged) and remove it from the Polaris > >>> YAML file. The API definitions remain the same—only their location > >> changes. > >>> The downside of this approach is that it introduces additional > references > >>> across multiple YAML files. > >>> > >>> Would love to hear your thoughts on this! > >>> > >>> Best regards, > >>> Honah > >>> > >>> [1] https://github.com/apache/polaris/pull/856 > >>> [2] > >>> > >> > https://techdocs.studio/blog/how-to-split-open-api-spec-into-multiple-files > >>> > >>> On Thu, Jan 23, 2025 at 2:22 AM Jean-Baptiste Onofré <j...@nanthrax.net> > >>> wrote: > >>> > >>>> Hi Jack, > >>>> > >>>> Welcome back! > >>>> > >>>> I like your idea to separate Iceberg REST Apis and Polaris REST Apis. > >>>> It's cleaner and it will certainly simplify the maintenance/update > >>>> (especially for the Iceberg REST APIs). > >>>> > >>>> About the Polaris REST APIs, that's the idea: we start first in > >>>> Polaris but there is good chance to be proposed to Iceberg REST Spec. > >>>> As the "pace" between implementation and spec is different, the > >>>> purpose here is to start with an implementation and propose a generic > >>>> spec later. > >>>> > >>>> I think we should also cleanly state the "scope" of the APIs related > >>>> to prefix. Imho, we are gathering too much APIs in /catalog and > >>>> /management: I think we should consider more fine-grained prefix > >>>> and/or inner prefix, else we may have to create a bunch of routes from > >>>> one prefix to another (when an API move from Polaris to Iceberg etc). > >>>> > >>>> Regards > >>>> JB > >>>> > >>>> On Wed, Jan 22, 2025 at 10:39 PM Yufei Gu <flyrain...@gmail.com> > >> wrote: > >>>>> Thanks, Jack, for bringing this up! I think separating the Iceberg > >> REST > >>>>> APIs from the other Polaris REST APIs is an excellent idea. Not only > >>> does > >>>>> it simplify any potential future rebases of the Iceberg REST spec, > >> but > >>> it > >>>>> also provides developers with a clearer distinction between the > >>> different > >>>>> sets of REST APIs. > >>>>> > >>>>> To implement this, we may need to experiment with the auto-generation > >>>>> tools. Feel free to file issues, or propose ideas. > >>>>> > >>>>> Yufei > >>>>> > >>>>> > >>>>> On Wed, Jan 22, 2025 at 1:30 PM Jack Ye <yezhao...@gmail.com> wrote: > >>>>> > >>>>>> Hi everyone, > >>>>>> > >>>>>> I am starting to work on Polaris related tasks, and realize that > >>>> currently > >>>>>> the state of the spec folder [1] in the project is a bit > >>> disorganized: > >>>>>> The Iceberg REST catalog (IRC) spec [2] contains non-standard APIs > >>>> like the > >>>>>> /notifications API, but misses a few APIs like the scan planning > >>> ones. > >>>>>> There is the Polaris management service API spec [3] that seems to > >>>> record > >>>>>> the identity management APIs in Polaris and will never be a part of > >>>> the IRC > >>>>>> spec. But I see new proposals like policy [4] and volume [5] that > >>> will > >>>>>> likely introduce new APIs, and I am not sure where those APIs would > >>> go, > >>>>>> will they be in separate OpenAPI YAML files, or a part of the > >>>> management > >>>>>> service APIs, or do we expect them to be eventually a part of the > >>>> official > >>>>>> IRC spec? > >>>>>> > >>>>>> I propose that we should agree upon a consistent framework to deal > >>> with > >>>>>> these APIs, to allow users to more clearly understand different > >> types > >>>> of > >>>>>> APIs in Polaris, and allow developers to more easily and more > >>>> consistently > >>>>>> add new ones or update existing ones. The framework I have in mind > >> is > >>>> as > >>>>>> the following: > >>>>>> > >>>>>> 1. there will be in general 2 types of APIs: IRC APIs and Polaris > >>>> APIs. The > >>>>>> IRC APIs are precisely following the IRC spec, whereas the Polaris > >>>> APIs are > >>>>>> only available in a Polaris server. It is possible for some Polaris > >>>> APIs to > >>>>>> be developed first to unblock support of certain use cases, and > >> later > >>>>>> contributed back to the IRC spec, depending on the appetite of both > >>>>>> communities. If that happens, the Polaris API will be deprecated in > >>>> favor > >>>>>> of the official IRC API when it gets accepted in Iceberg. > >>>>>> > >>>>>> 2. For Polaris APIs, it is subdivided into different groups by > >>>>>> functionality, and these groups are physically separated into > >>> different > >>>>>> YAML files. Currently there are 2 groups: > >>>>>> - Polaris management service > >>>>>> - Polaris notification service > >>>>>> > >>>>>> And there can be new groups in the future for policy or volume if > >>> those > >>>>>> proposals get accepted in Polaris. > >>>>>> > >>>>>> What do we think? > >>>>>> > >>>>>> Best, > >>>>>> Jack Ye > >>>>>> > >>>>>> [1] https://github.com/apache/polaris/tree/main/spec > >>>>>> [2] > >>>>>> > >> > https://github.com/apache/polaris/blob/main/spec/rest-catalog-open-api.yaml > >>>>>> [3] > >>>>>> > >>>>>> > >> > https://github.com/apache/polaris/blob/main/spec/polaris-management-service.yml > >>>>>> [4] > >>>>>> > >>>>>> > >> > https://docs.google.com/document/d/1kIiVkFFg9tPa5SH70b9WwzbmclrzH3qWHKfCKXw5lbs/edit?tab=t.0#heading=h.nly223xz13km > >>>>>> [5] > >>>>>> > >>>>>> > >> > https://docs.google.com/document/d/1ofljkrtiXRWc-v6hfkg_laKlYltepTPX7zsg44Tb-BY/edit?tab=t.0#heading=h.7ic5c343eju1 > > -- > Robert Stupp > @snazy > >
