Small question, why catalog*s*? On Tue, Feb 25, 2025 at 5:51 AM Jean-Baptiste Onofré <j...@nanthrax.net> wrote:
> Hi folks, > > I created the https://github.com/apache/polaris-catalogs-migrator > repository. > > I will work with Ajantha to populate it. > > Regards > JB > > On Thu, Feb 20, 2025 at 12:33 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > > OK, let's move forward then > > > > I will prepare https://github.com/apache/polaris-catalogs-migrator > > repository as a transition phase. > > > > > I also support inviting Ajantha as a committer. > > That will be a separate discussion with PPMC. > > > > > Still really prefer a separate repository, at least for now. We can > always merge later. It also makes the "migration to ASF" easier. > > > > > > The "Nessie Iceberg-catalog-migrator tool" has been built to support > migrations from any catalog to any other catalog. This means, that the test > matrix is quite complex and I expect it to become even more complex and > time consuming. IMHO there's no need to "bother" Polaris "main CI" for > every PR with catalog-migrator tests. I would also expect a different > release cycle - no need to tie both together. > > > > Understood, let's use a specific repo for now. > > > > > Better use org.apache.polaris.catalogmigrator as the base group ID. > > > > Yes, that was the intent indeed. > > > > > This requires https://github.com/apache/polaris/pull/785. I'd also > prefer to keep the ITs against Nessie and add Polaris. > > > > The itests will probably evolve a lot depending of the new use of > > catalogs-migrator. > > The itests bring a bunch of dependencies (Nessie, Hive, ...). As we > > are talking about test dependencies, that's OK. > > > > > > > > We don't know yet how these (and other features) will look like and > not how it'll be related to the donated tool. > > > > > > > I'm very enthusiastic about catalogs-migrator: I see a lot of potential > :) > > For instance, the first "obvious" move is probably catalogs-migrator > > will evolve from a "standalone tool" to a library that we can use in > > Polaris server (federated catalog, etc). > > > > > > > > Please coordinate the migration from the source repo with me. We need > to clean some things up on the projectnessie Github org side first. > > > > > > > Ack, I will ping you :) > > > > For the rest of the community, no objection to starting with > > https://github.com/apache/polaris-catalogs-migrator ? > > > > Regards > > JB > > > > > > > > Robert > > > > > > On 20.02.25 09:57, Jean-Baptiste Onofré wrote: > > > > Hi Dmitri > > > > > > > > About "evolution plan", I see the catalog migrator tool evolving as a > > > > set of beans/providers that will be used in both CLI, and some server > > > > features (like federated catalogs or "foreign catalogs"). > > > > We should not focus too much on catalog migrator as it is today but > > > > more how it will be tomorrow. > > > > > > > > That's why I'm more in favor of preparing the field and donating as a > > > > module in the Polaris repo. > > > > > > > > Regards > > > > JB > > > > > > > > On Thu, Feb 20, 2025 at 6:38 AM Dmitri Bourlatchkov < > di...@apache.org> wrote: > > > >> +1 to accept the catalog migrator tool. > > > >> > > > >> I support inviting Ajantha as a committer. > > > >> > > > >> As to the source location, I tend to think that a separate repo > makes sense > > > >> with the current state of the code, but I also agree that the > overhead of > > > >> that may be too much, given that the codebase is small. I'm fine > with > > > >> either a separate repo or a new module in the current Polaris repo. > > > >> > > > >> What is the general plan for the evolution of the migrator tool? > Are we > > > >> talking about integrating it into Polaris Servers or will it remain > a > > > >> standalone tool as it is now? > > > >> > > > >> Thanks, > > > >> Dmitri. > > > >> > > > >> On Wed, Feb 19, 2025 at 11:39 AM Jean-Baptiste Onofré < > j...@nanthrax.net> > > > >> wrote: > > > >> > > > >>> Hi folks, > > > >>> > > > >>> Let me try to sum-up this topic. > > > >>> > > > >>> 1. Catalog Migration landing > > > >>> It seems we have a preference to land catalog-migrator as a module > on > > > >>> the main polaris repo. > > > >>> Robert expressed comments about CI, release cycle, dependencies. > > > >>> > > > >>> My view on that is that the purpose of the catalog-migrator is to > > > >>> evolve, and could become a key component for features like > federated > > > >>> catalogs. > > > >>> Due to that, I think we can consider catalog-migrator as a module, > > > >>> integrated in the Polaris CLI, or in the Polaris server, > > > >>> > > > >>> Robert, does it work for you ? > > > >>> > > > >>> 2. Code/PR prep > > > >>> I propose to work directly with Ajantha (main contributor of the > > > >>> catalog-migrator) to prepare the code heading to a PR. We need: > > > >>> - integrate in Polaris repo and gradle > > > >>> - rename all packages to use org.apache.polaris > > > >>> - add ASF header in all files > > > >>> - refactore cli to use polaris style/naming > > > >>> - refactore intTest to use Polaris instead of Nessie > > > >>> - check the dependencies in the cli uber jar (hadoop, hive, ...) > and > > > >>> cleanup LICENSE/NOTICE there > > > >>> - update README and cleanup other files > > > >>> It should be pretty fast and we should be able to create a PR for > > > >>> review/donation. > > > >>> > > > >>> Thoughts ? > > > >>> > > > >>> Regards > > > >>> JB > > > >>> > > > >>> On Tue, Feb 11, 2025 at 7:45 PM Jean-Baptiste Onofré < > j...@nanthrax.net> > > > >>> wrote: > > > >>>> Hi folks, > > > >>>> > > > >>>> As discussed with some of you, we would like to propose donating > the > > > >>>> Nessie Iceberg Catalog migrator tool > > > >>>> (https://github.com/projectnessie/iceberg-catalog-migrator) to > Apache > > > >>>> Polaris. > > > >>>> > > > >>>> A SGA has been already signed in case we accept the donation. > > > >>>> > > > >>>> In terms of donation, I propose the following: > > > >>>> - the iceberg-catalog-migration can land in a separate Polaris > repo > > > >>>> (as it is today for Nessie): > > > >>>> https://github.com/apache/polaris-catalog-migration or land as a > > > >>>> module in polaris repo directly > > > >>>> - we need to prepare the donation by changing the package names, > etc > > > >>>> - I would suggest to consider inviting one of the main > contributor of > > > >>>> Catalog Migrator (ajantha-bhat) as Polaris committer > > > >>>> > > > >>>> WDYT ? Do we accept the iceberg-catalog-migrator tool in Polaris > (we > > > >>>> can do a formal vote if we don't have obvious consensus) ? > > > >>>> > > > >>>> Regards > > > >>>> JB > > > > > > -- > > > Robert Stupp > > > @snazy > > > >
