IMO, anything that's not directly managed by Polaris source code is external. If someone can create a table directly in the other catalog and it shows up in Polaris, it's external.
As of now, all of the direct manipulation APIs are blocked for external catalogs - createTable, updateTable, etc. I think that's something we can change so that if we *can* delegate to a remote catalog, we should. In such a case, the privileges defined in Polaris should be applied (though, of course, we can't guarantee that they're applied by the remote catalog). Credential vending rules should apply according to the External catalog rules - if they're disabled for external catalogs, they can be enabled on a per-catalog basis. "migrate" would be a cool feature, but I'd put it low on the priority list. Those are my thoughts, anyway. On Thu, Oct 31, 2024 at 9:22 AM Russell Spitzer <russell.spit...@gmail.com> wrote: > Hi Y’all, > > Some of us at Snowflake and Revolut have been talking a bit about how > Apache Polaris can be used in conjunction with older implementations of > Apache Iceberg Catalogs. At Revolut, they have already built a version of > this which allows them to use Polaris Capabilities on top of an old Catalog > implementation and give them a migration path towards moving towards a 100% > Polaris solution in the future. Those of us at Snowflake were thinking that > this would be a great capability for Polaris to natively support so we were > hoping to bring up the topic with the greater Apache Polaris community to > see if other folks are interested in this and to help us sus out what > questions we need to answer to fit into everyone’s use cases. > > To start off the conversation here are a few questions I think we need to > answer: > > 1. Do we consider Iceberg Catalogs wrapped by the Polaris catalog as > External or part of Polaris? > 2. What should we pull to Polaris for policy application? Namespace, > Tables? How often should this get listed? > 3. Should we attempt any type of credential forwarding or should we just > assume Polaris gets full access to the wrapped Catalog? > 4. Should we provide a “migrate” functionality that allows someone to “end” > the federation and move to a solely Polaris architecture? > > I’m sure there are more but I want to get the ball rolling so the folks at > Revolut can work on a proposal with the community already somewhat in sync > about requirements/ideas. I'm linking to a proposal > < > https://docs.google.com/document/d/17GPe_qawoVlEJe3qwGKgMJlx2CYU97-Jz_npTy_b8_k/edit?tab=t.0#heading=h.nuhm7bltkglt > > > from @Almaz Galiev <almaz.gal...@revolut.com> to start thinking about this > > > Thanks for your time as always, > Russ >
