Re: Introducing Catalogs to Beam SQL

2025-06-16 Thread Kenneth Knowles
Aside about the annotations: we used to have @Experimental annotations but users didn't actually notice and do things differently, mostly, and we always forgot to "graduate" things to stable, so we removed them. I agree with the intention though - it would be nice to let people know that this is ne

Re: Introducing Catalogs to Beam SQL

2025-06-16 Thread Kenneth Knowles via dev
Regarding the doc and CATALOG syntax: I like it. Added a couple comments. Kenn On Mon, Jun 16, 2025 at 10:39 AM Kenneth Knowles wrote: > Aside about the annotations: we used to have @Experimental annotations but > users didn't actually notice and do things differently, mostly, and we > always f

Re: Introducing Catalogs to Beam SQL

2025-06-16 Thread Jan Lukavský
Hi Ahmed, yes, essentially. Better would be something like @Unstable, but we don't have such annotation, currently. So the @Internal one must do.  Jan On 6/13/25 20:58, Ahmed Abualsaud via dev wrote: Yeah makes sense, I managed to annotate the new Catalog and CatalogManager interfaces with "

Re: Introducing Catalogs to Beam SQL

2025-06-13 Thread Ahmed Abualsaud via dev
Yeah makes sense, I managed to annotate the new Catalog and CatalogManager interfaces with "@Internal" before the release was cut. Is that what y'all had in mind? Best, Ahmed On Fri, Jun 13, 2025 at 8:59 AM Jan Lukavský wrote: > Hi Ahmed, > > thanks for such a quick write-up. This is a pretty g

Re: Introducing Catalogs to Beam SQL

2025-06-13 Thread Jan Lukavský
Hi Ahmed, thanks for such a quick write-up. This is a pretty good start! I left some comments, but if we have time pressure, I think we can release "something", but clearly mark it as experimental (or better unstable), so that users know what is the current state. WDYT?  Jan On 6/12/25 02:

Re: Introducing Catalogs to Beam SQL

2025-06-12 Thread Danny McCormick via dev
Thanks, the doc generally looks good to me. My suggestion here would be: 1) Move forward with the PR mostly as is (it seems like it has gotten meaningful feedback and has been merged, so I guess this is done). 2) Cut the release branch so we can start to make progress on releasing 3) In parallel m

Re: Introducing Catalogs to Beam SQL

2025-06-11 Thread Ahmed Abualsaud via dev
Hey Jan, thanks for calling that out. Ideally I would have liked to give more time for extensive engagement like we do with other proposals. The reason for the accelerated timeline on this PR is to pave the way for a "Getting Started" section for Beam on the Iceberg website, ideally before their 1

Re: Introducing Catalogs to Beam SQL

2025-06-11 Thread Talat Uyarer
Hi Ahmed, Thank you so much for this change. I have been waiting for these recent SQL changes for a while. Going forward, I agree with Jan about having a design doc to outline these changes. The underlying Java implementation is largely hidden from users, so that can be changed in the future

Re: Introducing Catalogs to Beam SQL

2025-06-11 Thread Jan Lukavský
Hi Ahmed, this is a great effort which is by no doubt greatly needed by the Beam project as a whole. On the other hand I think we should try to establish a way to pull the community into the discussion process. Could you sum up the the PR (not small) into a design document where we can have a