Using multiple YAML files per API / API area sounds good to me.
We should definitely use the Iceberg REST API YAML without any
modifications to avoid accidental mistakes.
Cheers,
Dmitri.
On Tue, Jan 28, 2025 at 2:28 AM Honah J. <hon...@apache.org> wrote:
Hi everyone,
Thanks for the proposal and all the great discussion! I also agree that
we
should separate Polaris-only APIs from the IRC spec to keep the IRC spec
consistent with the upstream version. I’d like to propose another way to
achieve this, as some APIs may not need a separate prefix but are still
better kept in separate YAML files.
This is particularly useful for cases where new APIs logically belong
under
`/catalog` (e.g., the notification API) but should remain separate from
the
IRC YAML file for practical reasons. Similar cases will likely arise in
the
future.
To address this, we can split the OpenAPI spec into multiple YAML files
while keeping them grouped under the same prefix. This allows
Polaris-specific APIs to stay separate while ensuring the Iceberg spec
remains untouched. I tested this approach using policy management APIs in
PR
#856 <https://github.com/apache/polaris/pull/856>[1], based on this blog
post
<
https://techdocs.studio/blog/how-to-split-open-api-spec-into-multiple-files
[2].
With this approach, when a Polaris catalog-level API is later contributed
to Iceberg, we don’t need to re-route anything. We simply pull the latest
version from IRC (assuming it is merged) and remove it from the Polaris
YAML file. The API definitions remain the same—only their location
changes.
The downside of this approach is that it introduces additional references
across multiple YAML files.
Would love to hear your thoughts on this!
Best regards,
Honah
[1] https://github.com/apache/polaris/pull/856
[2]
https://techdocs.studio/blog/how-to-split-open-api-spec-into-multiple-files
On Thu, Jan 23, 2025 at 2:22 AM Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:
Hi Jack,
Welcome back!
I like your idea to separate Iceberg REST Apis and Polaris REST Apis.
It's cleaner and it will certainly simplify the maintenance/update
(especially for the Iceberg REST APIs).
About the Polaris REST APIs, that's the idea: we start first in
Polaris but there is good chance to be proposed to Iceberg REST Spec.
As the "pace" between implementation and spec is different, the
purpose here is to start with an implementation and propose a generic
spec later.
I think we should also cleanly state the "scope" of the APIs related
to prefix. Imho, we are gathering too much APIs in /catalog and
/management: I think we should consider more fine-grained prefix
and/or inner prefix, else we may have to create a bunch of routes from
one prefix to another (when an API move from Polaris to Iceberg etc).
Regards
JB
On Wed, Jan 22, 2025 at 10:39 PM Yufei Gu <flyrain...@gmail.com>
wrote:
Thanks, Jack, for bringing this up! I think separating the Iceberg
REST
APIs from the other Polaris REST APIs is an excellent idea. Not only
does
it simplify any potential future rebases of the Iceberg REST spec,
but
it
also provides developers with a clearer distinction between the
different
sets of REST APIs.
To implement this, we may need to experiment with the auto-generation
tools. Feel free to file issues, or propose ideas.
Yufei
On Wed, Jan 22, 2025 at 1:30 PM Jack Ye <yezhao...@gmail.com> wrote:
Hi everyone,
I am starting to work on Polaris related tasks, and realize that
currently
the state of the spec folder [1] in the project is a bit
disorganized:
The Iceberg REST catalog (IRC) spec [2] contains non-standard APIs
like the
/notifications API, but misses a few APIs like the scan planning
ones.
There is the Polaris management service API spec [3] that seems to
record
the identity management APIs in Polaris and will never be a part of
the IRC
spec. But I see new proposals like policy [4] and volume [5] that
will
likely introduce new APIs, and I am not sure where those APIs would
go,
will they be in separate OpenAPI YAML files, or a part of the
management
service APIs, or do we expect them to be eventually a part of the
official
IRC spec?
I propose that we should agree upon a consistent framework to deal
with
these APIs, to allow users to more clearly understand different
types
of
APIs in Polaris, and allow developers to more easily and more
consistently
add new ones or update existing ones. The framework I have in mind
is
as
the following:
1. there will be in general 2 types of APIs: IRC APIs and Polaris
APIs. The
IRC APIs are precisely following the IRC spec, whereas the Polaris
APIs are
only available in a Polaris server. It is possible for some Polaris
APIs to
be developed first to unblock support of certain use cases, and
later
contributed back to the IRC spec, depending on the appetite of both
communities. If that happens, the Polaris API will be deprecated in
favor
of the official IRC API when it gets accepted in Iceberg.
2. For Polaris APIs, it is subdivided into different groups by
functionality, and these groups are physically separated into
different
YAML files. Currently there are 2 groups:
- Polaris management service
- Polaris notification service
And there can be new groups in the future for policy or volume if
those
proposals get accepted in Polaris.
What do we think?
Best,
Jack Ye
[1] https://github.com/apache/polaris/tree/main/spec
[2]
https://github.com/apache/polaris/blob/main/spec/rest-catalog-open-api.yaml
[3]
https://github.com/apache/polaris/blob/main/spec/polaris-management-service.yml
[4]
https://docs.google.com/document/d/1kIiVkFFg9tPa5SH70b9WwzbmclrzH3qWHKfCKXw5lbs/edit?tab=t.0#heading=h.nly223xz13km
[5]
https://docs.google.com/document/d/1ofljkrtiXRWc-v6hfkg_laKlYltepTPX7zsg44Tb-BY/edit?tab=t.0#heading=h.7ic5c343eju1
