OK, let's move forward then I will prepare https://github.com/apache/polaris-catalogs-migrator repository as a transition phase.
> I also support inviting Ajantha as a committer. That will be a separate discussion with PPMC. > Still really prefer a separate repository, at least for now. We can always > merge later. It also makes the "migration to ASF" easier. > > The "Nessie Iceberg-catalog-migrator tool" has been built to support > migrations from any catalog to any other catalog. This means, that the test > matrix is quite complex and I expect it to become even more complex and time > consuming. IMHO there's no need to "bother" Polaris "main CI" for every PR > with catalog-migrator tests. I would also expect a different release cycle - > no need to tie both together. Understood, let's use a specific repo for now. > Better use org.apache.polaris.catalogmigrator as the base group ID. Yes, that was the intent indeed. > This requires https://github.com/apache/polaris/pull/785. I'd also prefer to > keep the ITs against Nessie and add Polaris. The itests will probably evolve a lot depending of the new use of catalogs-migrator. The itests bring a bunch of dependencies (Nessie, Hive, ...). As we are talking about test dependencies, that's OK. > > We don't know yet how these (and other features) will look like and not how > it'll be related to the donated tool. > I'm very enthusiastic about catalogs-migrator: I see a lot of potential :) For instance, the first "obvious" move is probably catalogs-migrator will evolve from a "standalone tool" to a library that we can use in Polaris server (federated catalog, etc). > > Please coordinate the migration from the source repo with me. We need to > clean some things up on the projectnessie Github org side first. > Ack, I will ping you :) For the rest of the community, no objection to starting with https://github.com/apache/polaris-catalogs-migrator ? Regards JB > > Robert > > On 20.02.25 09:57, Jean-Baptiste Onofré wrote: > > Hi Dmitri > > > > About "evolution plan", I see the catalog migrator tool evolving as a > > set of beans/providers that will be used in both CLI, and some server > > features (like federated catalogs or "foreign catalogs"). > > We should not focus too much on catalog migrator as it is today but > > more how it will be tomorrow. > > > > That's why I'm more in favor of preparing the field and donating as a > > module in the Polaris repo. > > > > Regards > > JB > > > > On Thu, Feb 20, 2025 at 6:38 AM Dmitri Bourlatchkov <di...@apache.org> > > wrote: > >> +1 to accept the catalog migrator tool. > >> > >> I support inviting Ajantha as a committer. > >> > >> As to the source location, I tend to think that a separate repo makes sense > >> with the current state of the code, but I also agree that the overhead of > >> that may be too much, given that the codebase is small. I'm fine with > >> either a separate repo or a new module in the current Polaris repo. > >> > >> What is the general plan for the evolution of the migrator tool? Are we > >> talking about integrating it into Polaris Servers or will it remain a > >> standalone tool as it is now? > >> > >> Thanks, > >> Dmitri. > >> > >> On Wed, Feb 19, 2025 at 11:39 AM Jean-Baptiste Onofré <j...@nanthrax.net> > >> wrote: > >> > >>> Hi folks, > >>> > >>> Let me try to sum-up this topic. > >>> > >>> 1. Catalog Migration landing > >>> It seems we have a preference to land catalog-migrator as a module on > >>> the main polaris repo. > >>> Robert expressed comments about CI, release cycle, dependencies. > >>> > >>> My view on that is that the purpose of the catalog-migrator is to > >>> evolve, and could become a key component for features like federated > >>> catalogs. > >>> Due to that, I think we can consider catalog-migrator as a module, > >>> integrated in the Polaris CLI, or in the Polaris server, > >>> > >>> Robert, does it work for you ? > >>> > >>> 2. Code/PR prep > >>> I propose to work directly with Ajantha (main contributor of the > >>> catalog-migrator) to prepare the code heading to a PR. We need: > >>> - integrate in Polaris repo and gradle > >>> - rename all packages to use org.apache.polaris > >>> - add ASF header in all files > >>> - refactore cli to use polaris style/naming > >>> - refactore intTest to use Polaris instead of Nessie > >>> - check the dependencies in the cli uber jar (hadoop, hive, ...) and > >>> cleanup LICENSE/NOTICE there > >>> - update README and cleanup other files > >>> It should be pretty fast and we should be able to create a PR for > >>> review/donation. > >>> > >>> Thoughts ? > >>> > >>> Regards > >>> JB > >>> > >>> On Tue, Feb 11, 2025 at 7:45 PM Jean-Baptiste Onofré <j...@nanthrax.net> > >>> wrote: > >>>> Hi folks, > >>>> > >>>> As discussed with some of you, we would like to propose donating the > >>>> Nessie Iceberg Catalog migrator tool > >>>> (https://github.com/projectnessie/iceberg-catalog-migrator) to Apache > >>>> Polaris. > >>>> > >>>> A SGA has been already signed in case we accept the donation. > >>>> > >>>> In terms of donation, I propose the following: > >>>> - the iceberg-catalog-migration can land in a separate Polaris repo > >>>> (as it is today for Nessie): > >>>> https://github.com/apache/polaris-catalog-migration or land as a > >>>> module in polaris repo directly > >>>> - we need to prepare the donation by changing the package names, etc > >>>> - I would suggest to consider inviting one of the main contributor of > >>>> Catalog Migrator (ajantha-bhat) as Polaris committer > >>>> > >>>> WDYT ? Do we accept the iceberg-catalog-migrator tool in Polaris (we > >>>> can do a formal vote if we don't have obvious consensus) ? > >>>> > >>>> Regards > >>>> JB > > -- > Robert Stupp > @snazy >
