Hello Apache XTable (Incubating) Community,

This is a discussion regarding a new feature request I have created in GH.
https://github.com/apache/incubator-xtable/issues/590

*Context*
Users of Apache XTable (Incubating) today can translate metadata across
table formats (iceberg, hudi, and delta) and use the tables in different
platforms depending on their choice. Today there's still some friction
involved in terms of usability because users need to explicitly register
the tables <https://xtable.apache.org/docs/catalogs-index/> in the catalog
of their choice (glue, HMS, unity, bigLake etc.) and then use the catalog
in the platform of their choice to do DDL, DML queries.

XTable is built on the principle of omni directional interoperability and
I'm proposing an interface which allows syncing metadata of table formats
to multiple catalogs in a continuous and incremental manner.


*Why do we need this feature ?*1. Reduce friction for XTable users - XTable
sync will register the tables in the catalogs of their choice after
metadata generation. If users are using a single format, they can still use
XTable to sync the metadata across multiple catalogs.
2. Avoid catalog lock-in - There's no reason why data/metadata in storage
should be registered in a single catalog, users can register the table
across multiple catalogs depending on the use-case, ecosystem and features
provided by the catalog.

*Implementation*
I have submitted a PR with the interfaces for CatalogSyncClient and
CatalogSyncOperations,
https://github.com/apache/incubator-xtable/pull/591

Any inputs/feedback from the community who are interested in collaborating
on the design and implementation of this feature to respond to this email
or join the discussion directly on GitHub. Your input, whether in design
suggestions or implementation support, would be appreciated.

Thanks
Vinish

Reply via email to