+1 supportive of this as well. We've long held this view in the Apache Hudi community. XTable can bring those to other table formats as well now.
Curious to hear others' thoughts here.. P.S: should we create a discussion on GH as well - cross link dev thread for easy discoverability. On Sat, Dec 7, 2024 at 5:17 PM Kyle Weller <kywe...@gmail.com> wrote: > I also think this is a great idea, I just came from AWS re:Invent > conference this week and I heard many people bring up this need. I think > this is also aligned to the project's goals and intentions of bringing > cross-table interoperability since the metastore or catalog is key to how > you typically access open table formats. So whether you can sync the > metadata to 1 or to multiple catalogs, this simple task is sorely needed. > As more catalogs are starting to grow in the community: Unity Catalog, > Apache Polaris (incubating), Apache Gravitino, DataHub, etc, the need for > multiple catalogs may also grow. > > Thanks, > Kyle > > On Fri, Dec 6, 2024 at 12:16 AM Vinish Reddy <vin...@apache.org> wrote: > > > Hello Apache XTable (Incubating) Community, > > > > This is a discussion regarding a new feature request I have created in > GH. > > https://github.com/apache/incubator-xtable/issues/590 > > > > *Context* > > Users of Apache XTable (Incubating) today can translate metadata across > > table formats (iceberg, hudi, and delta) and use the tables in different > > platforms depending on their choice. Today there's still some friction > > involved in terms of usability because users need to explicitly register > > the tables <https://xtable.apache.org/docs/catalogs-index/> in the > catalog > > of their choice (glue, HMS, unity, bigLake etc.) and then use the catalog > > in the platform of their choice to do DDL, DML queries. > > > > XTable is built on the principle of omni directional interoperability and > > I'm proposing an interface which allows syncing metadata of table formats > > to multiple catalogs in a continuous and incremental manner. > > > > > > *Why do we need this feature ?*1. Reduce friction for XTable users - > XTable > > sync will register the tables in the catalogs of their choice after > > metadata generation. If users are using a single format, they can still > use > > XTable to sync the metadata across multiple catalogs. > > 2. Avoid catalog lock-in - There's no reason why data/metadata in storage > > should be registered in a single catalog, users can register the table > > across multiple catalogs depending on the use-case, ecosystem and > features > > provided by the catalog. > > > > *Implementation* > > I have submitted a PR with the interfaces for CatalogSyncClient and > > CatalogSyncOperations, > > https://github.com/apache/incubator-xtable/pull/591 > > > > Any inputs/feedback from the community who are interested in > collaborating > > on the design and implementation of this feature to respond to this email > > or join the discussion directly on GitHub. Your input, whether in design > > suggestions or implementation support, would be appreciated. > > > > Thanks > > Vinish > > >
