The current /catalog/v1 URI prefix encapsulates the Iceberg REST Catalog
API. This API is not owned by Polaris.
If Polaris were to add new endpoints under /catalog/v1 and then Iceberg
releases a v2 REST Catalog API what happens to the Polaris-specific
endpoints? What if Iceberg adds an endpoint that matches the Polaris
endpoint URI (unlikely but possible)?
From my POV this is the primary reason for putting the new API under a
different prefix.
A secondary concern is that listing methods in the Iceberg REST API will
certainly not include entities created via the proposed new API, so mixing
new create endpoints with old lising endpoints does not make sense to me
from the REST API design perspective.
As for a practical proposal, how about the following?
* /catalog/v1 - remains supported for Iceberg REST API v1, but deprecated
and will not be used for v2.
* /catalog/iceberg/ - encapsulates the Iceberg REST API v1 (and future API
versions).
* /catalog/generic/v1 - the API proposed in the referenced document.
* /management - current Polaris Management API.
Now, "generic" is, of course, a naming concern. Other opinions are welcome
regarding how to call the new API.
Options that look reasonable to me (in order of preference):
* /catalog/polaris/v1 - this implies that Polaris evolves this API as the
primary API for all things catalogued by Polaris (not including roles,
grants, etc., which fall under /management)
* /catalog/api/v1 - same as above, but different path segment
* /catalog/storage-object/v1
* /catalog/generic/v1 - Works fine for the current proposal, but it will
look awkward if we have to add typed / specialized endpoints later (so
those API extensions may need yet another prefix).
* /catalog/generic-table/v1 - same as above
* /catalog/storage-entity/v1
Cheers,
Dmitri.
On Mon, Feb 24, 2025 at 3:10 PM yun zou <yunzou.colost...@gmail.com>
wrote:
Hi All,
Thanks all for attending the review today and providing your valuable
feedback, I really appreciate it ! And apologize for the late notice for
the review meeting here.
I think we had a good discussion, and agreed to
1) Move on to provide a new Spark Catalog Plugin and new set of APIs with
restricted scope to table managements, but keep the doors open for future
extensions includes new fields or even new APIs.
2) Update the field create_at_ms to catalog_register_at to be more clear
that the timestamp is for the registration time with Polaris.
We will start working on figuring out more implementation details for the
plugins as well as refactoring needed for Apache Polaris. If it turns out
too complicated, we can have a separate review or discussion about the
implementation part also.
As a follow up of the meeting, we would like to open this thread for
discussion for the URI prefix for the new endpoint. As Dmitri brought up
on
the design doc, since this set of APIs are new and not restricted by
Iceberg anymore, shall we introduce a new prefix like
/polaris/catalog/v1/
instead of reuse the current /catalog/v1?
I think my concern about this proposal is that it seems to diverge how
the
APIs path are formalized today, the APIs in Polaris today use a path that
is prefixed by functionality, like /catalog/ or /management/. If we start
using /polaris/catalog/, shall we change the /management also to
/polaris/management ? Shall we move the new Policy Engine related APIs to
/polaris/catalog also?
Another choice we have is to use /polaris-catalog/v1 instead /catalog/v1
as
a prefix, which aligns with the current API path better. Then we have the
same question, what do we do with the Policy Engine APIs? The Policies
can
eventually be applied to both table entities created through the /catalog
APIs and /polaris/catalog APIs.
Overall, I would prefer to stay consistent with the current APIs and
continue to use /catalog/v1, since regardless of Iceberg or Generic
tables,
they are just entities managed by Polaris, and the APIs don't have
'iceberg' in the path also. I would like to hear what people think about
this?
Link to the doc:
https://docs.google.com/document/d/1_R9jBIwoH3CV9G7gSoRJPQVEcOsROp4IEiGoeYeQE8A/edit?pli=1&tab=t.0
Best Regards,
Yun
