Thanks for sharing the updates. +1 for the plan (start with a `beam run` option and 'beam pipe' option, finally 'beam sql'). For "beam-sql", there is also an existing IPython/JupyterLab "magic" supported in Beam:
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/sql/beam_sql_magics.py similar to that `beam run` invokes `yaml_main`. Basically one can use `%beam_sql <parameters>` in a CLI fashion, but it preserves state (pcollections) across cells in a notebook session. Regards, Yi On Mon, Nov 2, 2026 at 10:47 AM Jignesh Shah via dev <[email protected]> wrote: > Over the weekend, I think I prototyped a way to combine both beam pipe' > and 'beam-sql' together using 'beam sql' command option (equivalent to the > beam-sql.sh script). It does create a dependency to have both Python and > Java installed but can be done together as a unified CLI if both languages > are installed. Right now my plan is to start with a `beam run` option to > set up the configuration in the code, then incorporate the 'beam pipe' > option, and finally 'beam sql'. There is also an ease-of-use feature, `beam > config`, to simplify common setup across runners, such as selecting a > default runner so you do not have to repeat options in every command. This > is something I thought was useful because I was repeating the tests to try > specific runner configurations, which reduced the required typing. > > Regards, > Jignesh > > > On Fri, Oct 30, 2026 at 1:34 PM Chamikara Jayalath <[email protected]> > wrote: > >> +1 for trying this out as well. I think, ideally, we should have one >> command though. The boundary between simple and complex pipelines can vary >> based on user expectations, and it's probably best not to make the more >> permanent command line APIs depend on that distinction. How about just >> using "beam run" for both use-cases but defaulting to the current >> definition of "beam pipe" if an argument for the pipeline file, say >> "--pipeline" is not provided? >> >> Thanks, >> Cham >> >> >> On Fri, Oct 30, 2026 at 11:52 AM Kenneth Knowles <[email protected]> wrote: >> >>> I'm a fan. I like that this offers some of the functionality that would >>> otherwise be in `gcloud dataflow` but for all Beam users. I also like the >>> zero-deps first pipeline. >>> >>> I'm generally of the belief that now is a time where we should "just try >>> stuff" and see what gets used. As long as it doesn't incur a big >>> maintenance burden, and has a chance to be useful, I say go for it. (and >>> I'm confident the technical details will shake out) >>> >>> Kenn >>> >>> On Fri, Oct 30, 2026 at 7:55 AM Danny McCormick via dev < >>> [email protected]> wrote: >>> >>>> Thanks - could be worth trying, mostly because the cost of trying it is >>>> low. >>>> >>>> I will note that this ends up as a simpler, but also less powerful, >>>> version of https://beam.apache.org/documentation/dsls/sql/shell/. It >>>> may be worth looking for a way to unify those concepts a little bit more. >>>> >>>> Any thoughts on the value add of this CLI vs the existing one? >>>> >>>> > Should a beam console script live in the core apache-beam package, or >>>> in a separate package? >>>> >>>> Putting it in apache-beam seems fine since it won't carry other >>>> dependencies AFAIK >>>> >>>> > Is the scope right for a first step: Python only, Beam YAML only, >>>> files plus Kafka and Pub/Sub for auto streaming? >>>> >>>> I don't understand this question - aren't we supporting all the IOs >>>> that Yaml supports? We're also not directly interacting with Python (other >>>> than via Yaml), right? >>>> >>>> Thanks, >>>> Danny >>>> >>>> On Wed, Oct 28, 2026 at 6:41 PM Jignesh Shah via dev < >>>> [email protected]> wrote: >>>> >>>>> >>>>> Hi all, >>>>> >>>>> I'd like feedback on a design for a small beam command-line tool that >>>>> makes Beam YAML easier to try: >>>>> >>>>> Design doc: <Apache Beam CLI Design Proposal >>>>> <https://docs.google.com/document/d/1TKEeGGokrcOZhFIdUD-JKgkRfm9a2EYPBkaZyJlnHy4/edit?usp=sharing>, >>>>> comments enabled> >>>>> >>>>> *The idea in two commands* >>>>> >>>>> pip install 'apache-beam[yaml]' >>>>> beam pipe --from users.csv --to out/users.parquet --filter "age >= 21" >>>>> beam pipe ... --dry-run > pipeline.yaml # then edit it and: >>>>> beam run pipeline.yaml [--runner=...] >>>>> >>>>> beam run is a thin front end for python -m apache_beam.yaml.main: >>>>> same pipelines, same runners and pipeline options. beam pipe covers >>>>> the simplest case—moving data between files, Kafka and Pub/Sub topics with >>>>> an optional filter and field selection— and prints the equivalent YAML >>>>> with >>>>> --dry-run so users can grow it into a real pipeline. Both run on the >>>>> SDK's default runner (Prism, with its usual fallback) unless --runner is >>>>> given. No new IO or runner code is involved; it reuses Beam YAML's >>>>> existing >>>>> transforms. >>>>> >>>>> *Questions where I'd most value input* >>>>> >>>>> 1. Should a beam console script live in the core apache-beam package, >>>>> or in a separate package? >>>>> 2. Is the command name pipe right (alternatives in the doc)? >>>>> 3. Is the scope right for a first step: Python only, Beam YAML >>>>> only, files plus Kafka and Pub/Sub for auto streaming? >>>>> 4. For Spark and Flink maintainers: is uber-JAR submission the >>>>> path we want to recommend, or make docker a requirement for an easy >>>>> experience? >>>>> >>>>> I'd like to keep comments open for about two weeks before deciding >>>>> whether to proceed. If it goes ahead, the plan is a series of small PRs >>>>> rather than one large one. >>>>> >>>>> Disclosure: I used an AI coding assistant to help write the code >>>>> prototype and draft the design doc, which I then manually edited. >>>>> >>>>> Thanks, >>>>> >>>>> Jignesh >>>>> >>>>>
