Haven't read the proposal yet (so bear with me ;) )
Integrating other Iceberg catalogs is a great idea! However, should we
maybe concentrate only on linking to external REST based catalogs and
let the remote ones deal with "specialties" like how often to list
namespaces/tables/views and whether the results shall be memoized? This
would decouple the "core" Polaris service and likely prevent dependency
issues. Either it's a Polaris configured catalog that rather "just
proxies" requests or, more advanced, a "special namespace" in an
existing catalog that links to a remote catalog. WDYT?
On 31.10.24 17:21, Russell Spitzer wrote:
Hi Y’all,
Some of us at Snowflake and Revolut have been talking a bit about how
Apache Polaris can be used in conjunction with older implementations of
Apache Iceberg Catalogs. At Revolut, they have already built a version of
this which allows them to use Polaris Capabilities on top of an old Catalog
implementation and give them a migration path towards moving towards a 100%
Polaris solution in the future. Those of us at Snowflake were thinking that
this would be a great capability for Polaris to natively support so we were
hoping to bring up the topic with the greater Apache Polaris community to
see if other folks are interested in this and to help us sus out what
questions we need to answer to fit into everyone’s use cases.
To start off the conversation here are a few questions I think we need to
answer:
1. Do we consider Iceberg Catalogs wrapped by the Polaris catalog as
External or part of Polaris?
2. What should we pull to Polaris for policy application? Namespace,
Tables? How often should this get listed?
3. Should we attempt any type of credential forwarding or should we just
assume Polaris gets full access to the wrapped Catalog?
4. Should we provide a “migrate” functionality that allows someone to “end”
the federation and move to a solely Polaris architecture?
I’m sure there are more but I want to get the ball rolling so the folks at
Revolut can work on a proposal with the community already somewhat in sync
about requirements/ideas. I'm linking to a proposal
<https://docs.google.com/document/d/17GPe_qawoVlEJe3qwGKgMJlx2CYU97-Jz_npTy_b8_k/edit?tab=t.0#heading=h.nuhm7bltkglt>
from @Almaz Galiev <almaz.gal...@revolut.com> to start thinking about this
Thanks for your time as always,
Russ
--
Robert Stupp
@snazy
