Thanks, the doc generally looks good to me. My suggestion here would be: 1) Move forward with the PR mostly as is (it seems like it has gotten meaningful feedback and has been merged, so I guess this is done). 2) Cut the release branch so we can start to make progress on releasing 3) In parallel make sure we have consensus on the doc. Block actually releasing until we have that consensus (up to individual PMC members to dissent if that condition hasn't been reached). --- If we can't achieve consensus or it is different than the PR, either back out the PR or make cherry-picks as appropriate (one option would be to CP in a change to mark this as experimental for a release).
Hopefully we can iterate on the doc quickly if there are concerns so we can balance the long term goals against trying to make it in time for the 1.10 Iceberg release. Thanks, Danny On Wed, Jun 11, 2025 at 11:49 PM Ahmed Abualsaud via dev < dev@beam.apache.org> wrote: > Hey Jan, thanks for calling that out. Ideally I would have liked to give > more time for extensive engagement like we do with other proposals. > > The reason for the accelerated timeline on this PR is to pave the way for > a "Getting Started" section for Beam on the Iceberg website, ideally before > their 1.10.0 release cut on June 23rd. Integrating IcebergIO with Beam SQL > is crucial for reaching a wider audience, and getting this piece in before > the Beam release cut (today/tomorrow) would allow us to showcase SQL > support sooner. > > As Talat mentioned, the underlying Java implementation can be iterated on, > which means our immediate priority for consensus is the SQL syntax itself. > > I've put together an "after-the-fact" document to provide more context, > including a scouting report on how other frameworks handle catalog > management, and the proposed Beam SQL syntax. I hope this helps kickstart > the discussion. > > > https://docs.google.com/document/d/16P0JrcJ28KSoMMpLYExWPZaala7CE4Ezen-jC_ly3M4/edit?tab=t.0 > > Best, > Ahmed > > > On Wed, Jun 11, 2025 at 3:18 PM Talat Uyarer <ta...@apache.org> wrote: > >> Hi Ahmed, >> >> Thank you so much for this change. I have been waiting for these recent >> SQL changes for a while. >> >> Going forward, I agree with Jan about having a design doc to outline >> these changes. The underlying Java implementation is largely hidden from >> users, so that can be changed in the future, but as a community we should >> agree on the proposed SQL syntax. >> >> Jan, I am as a Beam user and a small contributor, I've also been waiting >> for this feature. And if you don't mind, can we get Ahmed's changes in this >> version? >> >> Thanks >> >> On 2025/06/11 18:42:40 Jan Lukavský wrote: >> > Hi Ahmed, >> > >> > this is a great effort which is by no doubt greatly needed by the Beam >> > project as a whole. On the other hand I think we should try to >> establish >> > a way to pull the community into the discussion process. Could you sum >> > up the the PR (not small) into a design document where we can have a >> > discussion about the goals, alternative solutions, already tried ways, >> > etc? This would be really cool! >> > >> > Best, >> > >> > Jan >> > >> > On 6/10/25 16:12, Ahmed Abualsaud via dev wrote: >> > > Hey all, >> > > >> > > I was integrating our Java IcebergIO with Beam SQL (PR #34799 >> > > <https://github.com/apache/beam/pull/34799>) and got blocked on the >> > > fact that Beam SQL currently lacks a "Catalog" concept. This is >> > > fundamental to modern data architectures like Iceberg, where they are >> > > used to manage table metadata and enable broad ecosystem integration. >> > > To address this gap, I've opened a new PR (#35223 >> > > <https://github.com/apache/beam/pull/35223>), which introduces the >> > > *Catalog* and *CatalogManager* interfaces, enabling support for: >> > > >> > > * >> > > >> > > |CREATE CATALOG my_catalog TYPE 'local' PROPERTIES (...)| >> > > >> > > * >> > > >> > > |SET CATALOG my_catalog| >> > > >> > > * >> > > >> > > |DROP CATALOG my_catalog| >> > > >> > > I left a more detailed overview in the PR description. >> > > >> > > My hope is that this foundational change will benefit not just >> > > IcebergIO, but also other IOs and future Beam SQL integrations. >> > > >> > > Please take a look and share any feedback, especially regarding major >> > > architectural concerns. I'm working on a short timeline, so minor >> > > enhancements can be noted for follow-up PRs. >> > > >> > > Thank you! >> > > Ahmed >> >
