Yes, the reason for not migrating/replicating from the metastore level was that it wasn't necessarily true that you had access to the metastore layer, but it was much more likely that there was some programmatic way to access the Polaris APIs.
On Fri, Apr 11, 2025 at 6:17 PM Eric Maynard <eric.w.mayn...@gmail.com> wrote: > For keeping two Polaris instances in sync, I agree that replicating at the > persistence layer probably makes the most sense. > > However there are cases when you want to copy data from one Polaris > instance to another but you may not have direct access to the metastore. > For example, migrating from a self-hosted Polaris instance to a managed > offering. To support these cases, I think a tool like this can be useful. > > On Fri, Apr 11, 2025 at 6:12 PM Ajantha Bhat <ajanthab...@gmail.com> > wrote: > > > Hey, Thanks for the proposal and I agree with Yufei. > > > > We had a backend synchronization CLI for projectNessie[1]. Maybe we can > > have something similar to that instead of taking a long path of the > > register table for migration between polaris instances. > > > > [1] https://projectnessie.org/nessie-0-82-0/export_import/ > > > > - Ajantha > > > > On Sat, Apr 12, 2025 at 5:54 AM Yufei Gu <flyrain...@gmail.com> wrote: > > > > > Thanks, Mansehaj, for the proposal! This tool has potential, but I > think > > we > > > should clarify its capabilities a bit more explicitly. Given its > current > > > limitations, I'm not sure how broadly useful it would be. Have we > > explored > > > any alternative approaches—for example, performing synchronization on > the > > > backend(FoundationDB, Postgres)? > > > > > > > > > Yufei > > > > > > On Thu, Apr 10, 2025 at 4:22 PM Mansehaj Singh > > > <mansehaj.si...@snowflake.com.invalid> wrote: > > > > > > > Hi all! Nice to meet you. > > > > > > > > I opened up https://github.com/apache/polaris-tools/pull/4 recently > to > > > add > > > > a Polaris migration/synchronizer tool I've been working on to the > > > > polaris-tools repo. By request, I'm sharing a design document here > > > > detailing how the tool works and the roadmap for functionality that > is > > in > > > > development. > > > > > > > > Here's the design doc giving a full overview: > > > > > > > > > > > > > > https://docs.google.com/document/d/1AXKmzp3JaTuUS_FMNnxr_pHsBTs86rWRMborMi3deCw/edit?usp=sharing > > > > > > > > > > > > To summarize: > > > > > > > > We can think of this tool as a configurable mirroring/migration tool > to > > > > migrate between two Polaris instances. I believe this would enable > and > > > > support many use cases that are quite cumbersome to carry out > manually > > > > today and break down barriers switching between open source and > managed > > > > offerings of Polaris. The tool has been designed with goals in mind > > that > > > go > > > > beyond supporting just the CLI implementation. > > > > > > > > Please take a look at the design doc if you're interested! > > > > > > > > Thank you! > > > > - Sehaj > > > > > > > > > >
