Hi Ahmed,
this is a great effort which is by no doubt greatly needed by the Beam
project as a whole. On the other hand I think we should try to establish
a way to pull the community into the discussion process. Could you sum
up the the PR (not small) into a design document where we can have a
discussion about the goals, alternative solutions, already tried ways,
etc? This would be really cool!
Best,
Jan
On 6/10/25 16:12, Ahmed Abualsaud via dev wrote:
Hey all,
I was integrating our Java IcebergIO with Beam SQL (PR #34799
<https://github.com/apache/beam/pull/34799>) and got blocked on the
fact that Beam SQL currently lacks a "Catalog" concept. This is
fundamental to modern data architectures like Iceberg, where they are
used to manage table metadata and enable broad ecosystem integration.
To address this gap, I've opened a new PR (#35223
<https://github.com/apache/beam/pull/35223>), which introduces the
*Catalog* and *CatalogManager* interfaces, enabling support for:
*
|CREATE CATALOG my_catalog TYPE 'local' PROPERTIES (...)|
*
|SET CATALOG my_catalog|
*
|DROP CATALOG my_catalog|
I left a more detailed overview in the PR description.
My hope is that this foundational change will benefit not just
IcebergIO, but also other IOs and future Beam SQL integrations.
Please take a look and share any feedback, especially regarding major
architectural concerns. I'm working on a short timeline, so minor
enhancements can be noted for follow-up PRs.
Thank you!
Ahmed
