Thanks for the reply, JB! You make good points and I agree that it would be great to achieve engine interoperability with Polaris.
Regarding security, I suppose that includes authenticating engines as catalog clients too. The Iceberg REST API spec does not appear to be very specific about that part (pardon the pun). To a large extent, client authentication becomes implementation specific with the Iceberg java REST client doing things in a certain way, while other clients may do things differently, or have to adjust to how the java client works. Iceberg PR 10753 [1] proposes a pluggable approach to authentication in the java client (perhaps being but step 1 in that direction). Having pluggable authentication in the java client should hopefully allow introducing well-defined authentication flows (e.g. OAuth2 device code) without impacting the client's stability when processing table metadata is concerned and without putting too much overhead on the Iceberg REST Catalog spec. The REST spec will not have to delve into the area of authentication, which happens to have existing standards already. I think that would also be beneficial for Polaris' interoperability goals. Hopefully when some standard mechanisms are available in the java client, other languages and (possibly) proprietary clients will adopt them too. I'm interested in hearing other people's thoughts on this matter too. Cheers, Dmitri. [1] https://github.com/apache/iceberg/pull/10753 On Thu, Oct 24, 2024 at 2:02 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > Hi Dmitri > > My take on "Polaris interoperability" is between query engines: any > query engine supporting the Iceberg REST API can use Polaris. The > purpose is to have an unique Polaris across query engines, in order to > centralize security. Today, in organizations, we often have several > query engines (for example, Flink for streaming engines, Spark for > transformation, Dremio for analytics, etc). So, it's important to have > an "interoperable" catalog usable by any engine. > Another take on "interoperability" is also between catalogs. As > Polaris supports external catalogs, we can see a kind of > interoperability between Polaris and other catalogs. > > It's my view, others might have another one (please share here :)). > > Maybe it's worth adding a section in the documentation to define > "Polaris interoperability". > > Regards > JB > > On Wed, Oct 23, 2024 at 11:23 PM Dmitri Bourlatchkov > <dmitri.bourlatch...@dremio.com.invalid> wrote: > > > > Hi All, > > > > The Polaris GH repo has this statement: "Apache Polaris, the > interoperable, > > open source catalog for Apache Iceberg". > > > > How is this "interoperability" defined? Do we have any docs that talk > about > > that? > > > > If not, what is the general perception in this community about what is > > considered interoperable and what is not? > > > > Thanks, > > Dmitri. >
