Export / Import does not have to be at the Persistence layer. If the Admin API is expressive enough, it should be possible use it instead, I guess.
Cheers, Dmitri. On Fri, Apr 11, 2025 at 9:17 PM Eric Maynard <eric.w.mayn...@gmail.com> wrote: > For keeping two Polaris instances in sync, I agree that replicating at the > persistence layer probably makes the most sense. > > However there are cases when you want to copy data from one Polaris > instance to another but you may not have direct access to the metastore. > For example, migrating from a self-hosted Polaris instance to a managed > offering. To support these cases, I think a tool like this can be useful. > > On Fri, Apr 11, 2025 at 6:12 PM Ajantha Bhat <ajanthab...@gmail.com> > wrote: > > > Hey, Thanks for the proposal and I agree with Yufei. > > > > We had a backend synchronization CLI for projectNessie[1]. Maybe we can > > have something similar to that instead of taking a long path of the > > register table for migration between polaris instances. > > > > [1] https://projectnessie.org/nessie-0-82-0/export_import/ > > > > - Ajantha > > > > On Sat, Apr 12, 2025 at 5:54 AM Yufei Gu <flyrain...@gmail.com> wrote: > > > > > Thanks, Mansehaj, for the proposal! This tool has potential, but I > think > > we > > > should clarify its capabilities a bit more explicitly. Given its > current > > > limitations, I'm not sure how broadly useful it would be. Have we > > explored > > > any alternative approaches—for example, performing synchronization on > the > > > backend(FoundationDB, Postgres)? > > > > > > > > > Yufei > > > > > > On Thu, Apr 10, 2025 at 4:22 PM Mansehaj Singh > > > <mansehaj.si...@snowflake.com.invalid> wrote: > > > > > > > Hi all! Nice to meet you. > > > > > > > > I opened up https://github.com/apache/polaris-tools/pull/4 recently > to > > > add > > > > a Polaris migration/synchronizer tool I've been working on to the > > > > polaris-tools repo. By request, I'm sharing a design document here > > > > detailing how the tool works and the roadmap for functionality that > is > > in > > > > development. > > > > > > > > Here's the design doc giving a full overview: > > > > > > > > > > > > > > https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing > > > > > > > > > > > > To summarize: > > > > > > > > We can think of this tool as a configurable mirroring/migration tool > to > > > > migrate between two Polaris instances. I believe this would enable > and > > > > support many use cases that are quite cumbersome to carry out > manually > > > > today and break down barriers switching between open source and > managed > > > > offerings of Polaris. The tool has been designed with goals in mind > > that > > > go > > > > beyond supporting just the CLI implementation. > > > > > > > > Please take a look at the design doc if you're interested! > > > > > > > > Thank you! > > > > - Sehaj > > > > > > > > > >
