I think it's really important that we have the functionality to move folks off of existing catalog solutions be they Hive, Hadoop or what not. I do think, even with the dependency issues, this should be a core functionality of Polaris. I think this would make it much easier for folks on non-REST catalogs to migrate to a REST solution. Even if we have a true "migrate" functionality punted till later, it would be great if we could provide at least a path towards REST completely within Polaris. I know that for us to federate with some systems like One Lake that don't have a REST endpoint, this will be a requirement.
On Thu, Oct 31, 2024 at 12:15 PM Michael Collado <collado.m...@gmail.com> wrote: > IMO, anything that's not directly managed by Polaris source code is > external. If someone can create a table directly in the other catalog and > it shows up in Polaris, it's external. > > As of now, all of the direct manipulation APIs are blocked for external > catalogs - createTable, updateTable, etc. I think that's something we can > change so that if we *can* delegate to a remote catalog, we should. In such > a case, the privileges defined in Polaris should be applied (though, of > course, we can't guarantee that they're applied by the remote catalog). > > Credential vending rules should apply according to the External catalog > rules - if they're disabled for external catalogs, they can be enabled on a > per-catalog basis. > > "migrate" would be a cool feature, but I'd put it low on the priority list. > > Those are my thoughts, anyway. > > On Thu, Oct 31, 2024 at 9:22 AM Russell Spitzer <russell.spit...@gmail.com > > > wrote: > > > Hi Y’all, > > > > Some of us at Snowflake and Revolut have been talking a bit about how > > Apache Polaris can be used in conjunction with older implementations of > > Apache Iceberg Catalogs. At Revolut, they have already built a version of > > this which allows them to use Polaris Capabilities on top of an old > Catalog > > implementation and give them a migration path towards moving towards a > 100% > > Polaris solution in the future. Those of us at Snowflake were thinking > that > > this would be a great capability for Polaris to natively support so we > were > > hoping to bring up the topic with the greater Apache Polaris community to > > see if other folks are interested in this and to help us sus out what > > questions we need to answer to fit into everyone’s use cases. > > > > To start off the conversation here are a few questions I think we need to > > answer: > > > > 1. Do we consider Iceberg Catalogs wrapped by the Polaris catalog as > > External or part of Polaris? > > 2. What should we pull to Polaris for policy application? Namespace, > > Tables? How often should this get listed? > > 3. Should we attempt any type of credential forwarding or should we just > > assume Polaris gets full access to the wrapped Catalog? > > 4. Should we provide a “migrate” functionality that allows someone to > “end” > > the federation and move to a solely Polaris architecture? > > > > I’m sure there are more but I want to get the ball rolling so the folks > at > > Revolut can work on a proposal with the community already somewhat in > sync > > about requirements/ideas. I'm linking to a proposal > > < > > > https://docs.google.com/document/d/17GPe_qawoVlEJe3qwGKgMJlx2CYU97-Jz_npTy_b8_k/edit?tab=t.0#heading=h.nuhm7bltkglt > > > > > from @Almaz Galiev <almaz.gal...@revolut.com> to start thinking about > this > > > > > > Thanks for your time as always, > > Russ > > >
