Hi Ahmed, Thank you so much for this change. I have been waiting for these recent SQL changes for a while.
Going forward, I agree with Jan about having a design doc to outline these changes. The underlying Java implementation is largely hidden from users, so that can be changed in the future, but as a community we should agree on the proposed SQL syntax. Jan, I am as a Beam user and a small contributor, I've also been waiting for this feature. And if you don't mind, can we get Ahmed's changes in this version? Thanks On 2025/06/11 18:42:40 Jan Lukavský wrote: > Hi Ahmed, > > this is a great effort which is by no doubt greatly needed by the Beam > project as a whole. On the other hand I think we should try to establish > a way to pull the community into the discussion process. Could you sum > up the the PR (not small) into a design document where we can have a > discussion about the goals, alternative solutions, already tried ways, > etc? This would be really cool! > > Best, > > Jan > > On 6/10/25 16:12, Ahmed Abualsaud via dev wrote: > > Hey all, > > > > I was integrating our Java IcebergIO with Beam SQL (PR #34799 > > <https://github.com/apache/beam/pull/34799>) and got blocked on the > > fact that Beam SQL currently lacks a "Catalog" concept. This is > > fundamental to modern data architectures like Iceberg, where they are > > used to manage table metadata and enable broad ecosystem integration. > > To address this gap, I've opened a new PR (#35223 > > <https://github.com/apache/beam/pull/35223>), which introduces the > > *Catalog* and *CatalogManager* interfaces, enabling support for: > > > > * > > > > |CREATE CATALOG my_catalog TYPE 'local' PROPERTIES (...)| > > > > * > > > > |SET CATALOG my_catalog| > > > > * > > > > |DROP CATALOG my_catalog| > > > > I left a more detailed overview in the PR description. > > > > My hope is that this foundational change will benefit not just > > IcebergIO, but also other IOs and future Beam SQL integrations. > > > > Please take a look and share any feedback, especially regarding major > > architectural concerns. I'm working on a short timeline, so minor > > enhancements can be noted for follow-up PRs. > > > > Thank you! > > Ahmed
