I agree with Ajentha's point. Treating this as export/import (via an intermediate format) will open more use cases to the admin user.
The original migration from a catalog to another catalog is still possible as a sub-case of export/import. Note: I made a comment on the doc about NOT exporting secrets. I think it still needs special attention in the export case. We probably still want to support full Polaris recovery with this tool, but the tool probably needs to expose special options for allowing the export of principal secrets. Even if they are only password hashes, exposing them in another environment will add security risks. Cheers, Dmitri. On Fri, Apr 11, 2025 at 9:11 PM Ajantha Bhat <ajanthab...@gmail.com> wrote: > Hey, Thanks for the proposal and I agree with Yufei. > > We had a backend synchronization CLI for projectNessie[1]. Maybe we can > have something similar to that instead of taking a long path of the > register table for migration between polaris instances. > > [1] https://projectnessie.org/nessie-0-82-0/export_import/ > > - Ajantha > > On Sat, Apr 12, 2025 at 5:54 AM Yufei Gu <flyrain...@gmail.com> wrote: > > > Thanks, Mansehaj, for the proposal! This tool has potential, but I think > we > > should clarify its capabilities a bit more explicitly. Given its current > > limitations, I'm not sure how broadly useful it would be. Have we > explored > > any alternative approaches—for example, performing synchronization on the > > backend(FoundationDB, Postgres)? > > > > > > Yufei > > > > On Thu, Apr 10, 2025 at 4:22 PM Mansehaj Singh > > <mansehaj.si...@snowflake.com.invalid> wrote: > > > > > Hi all! Nice to meet you. > > > > > > I opened up https://github.com/apache/polaris-tools/pull/4 recently to > > add > > > a Polaris migration/synchronizer tool I've been working on to the > > > polaris-tools repo. By request, I'm sharing a design document here > > > detailing how the tool works and the roadmap for functionality that is > in > > > development. > > > > > > Here's the design doc giving a full overview: > > > > > > > > > https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing > > > > > > > > > To summarize: > > > > > > We can think of this tool as a configurable mirroring/migration tool to > > > migrate between two Polaris instances. I believe this would enable and > > > support many use cases that are quite cumbersome to carry out manually > > > today and break down barriers switching between open source and managed > > > offerings of Polaris. The tool has been designed with goals in mind > that > > go > > > beyond supporting just the CLI implementation. > > > > > > Please take a look at the design doc if you're interested! > > > > > > Thank you! > > > - Sehaj > > > > > >
