Yeah it's true most of the stuff we're adding is standard SQL syntax. Most of the real work is just mapping it to a specific table format and the relevant connectors.
> Is there a way we can do this "all at once" ... or TBH integrating with something Well we're already integrated with Apache Calcite (similar to Flink), which gives us a lot of free parsing logic and object representation. Our Catalog/Database/Table representations all implement Calcite's Schema interface so they make use of Calcite's parsing, validation, planning. Beyond that, AFAIK every engine defines its own interface for applying DDL to a data source. For example Flink's Catalog API <https://nightlies.apache.org/flink/flink-docs-stable/api/java/org/apache/flink/table/catalog/Catalog.html> and Spark's TableCatalog API <https://spark.apache.org/docs/latest/api/java/org/apache/spark/sql/connector/catalog/TableCatalog.html>. >From what I know, there isn't a shared DDL mapping layer that we can plug into. > we shouldn't need many designs or discussion around it I think the underlying integration work is implementation detail and in that sense agree it shouldn't require a discussion. But I think syntax changes deserve at least a callout since SQL is such a high-level, user-facing language. There are also many devs here with stronger SQL experience and more insightful opinions than mine (for example, last thread's discussion turned the CHANGES clause into a function instead). Although maybe this time PRIMARY KEY is a no-brainer. Generally I'm trying to keep these proposals short and focused only on the syntax additions, with some comparison to other dialects. Best, Ahmed On Fri, Oct 23, 2026 at 10:48 AM Kenneth Knowles <[email protected]> wrote: > Seems clearly useful. > > Just to derail the thread... :-) what is the eventual endpoint of the Beam > SQL DDL work like this? It seems like we are probably duplicating something > that has been done many times before. Namely: having a model of what a > table is, and a way to map DDL to specific implementations that have > different backing connectors. > > I think it still makes sense to have our SQL CLI / driver or other things > built on Beam SQL be able to "do it all" because it is a whole-product > entrypoint. And of course many many sibling projects have their own full > implementation of all this stuff. > > Is there a way we can do this "all at once" and be done with it, having > the whole "you type SQL into Beam and we know how to do the thing to the > underlying storage / catalog" step? It feels like we shouldn't need many > designs or discussion around it, unless it has Beam-specific > considerations, and we could possibly crank it out semi-automatically? Or > TBH integrating with something is better than vibe coding, so as to get > maintenance and improvements for free. > > Kenn > > On Thu, Oct 22, 2026 at 11:33 PM Ahmed Abualsaud via dev < > [email protected]> wrote: > >> Hey everyone, >> >> Following up on the last proposal, this one allows users to create a >> table with a Primary Key or modify an existing table's Primary Key using >> Beam SQL. >> >> >> https://docs.google.com/document/d/1lGynrfHlyvcmmyzJZ2gkL9zK8KF1ZnfkcImnMmM_ciw/edit >> >> This is a necessary piece to facilitate using CDC in Beam SQL. Looking >> for feedback and suggestions! >> >> Best, >> Ahmed >> >
