Over the weekend, I think I prototyped a way to combine both beam pipe' and 'beam-sql' together using 'beam sql' command option (equivalent to the beam-sql.sh script). It does create a dependency to have both Python and Java installed but can be done together as a unified CLI if both languages are installed. Right now my plan is to start with a `beam run` option to set up the configuration in the code, then incorporate the 'beam pipe' option, and finally 'beam sql'. There is also an ease-of-use feature, `beam config`, to simplify common setup across runners, such as selecting a default runner so you do not have to repeat options in every command. This is something I thought was useful because I was repeating the tests to try specific runner configurations, which reduced the required typing.
Regards, Jignesh On Fri, Oct 30, 2026 at 1:34 PM Chamikara Jayalath <[email protected]> wrote: > +1 for trying this out as well. I think, ideally, we should have one > command though. The boundary between simple and complex pipelines can vary > based on user expectations, and it's probably best not to make the more > permanent command line APIs depend on that distinction. How about just > using "beam run" for both use-cases but defaulting to the current > definition of "beam pipe" if an argument for the pipeline file, say > "--pipeline" is not provided? > > Thanks, > Cham > > > On Fri, Oct 30, 2026 at 11:52 AM Kenneth Knowles <[email protected]> wrote: > >> I'm a fan. I like that this offers some of the functionality that would >> otherwise be in `gcloud dataflow` but for all Beam users. I also like the >> zero-deps first pipeline. >> >> I'm generally of the belief that now is a time where we should "just try >> stuff" and see what gets used. As long as it doesn't incur a big >> maintenance burden, and has a chance to be useful, I say go for it. (and >> I'm confident the technical details will shake out) >> >> Kenn >> >> On Fri, Oct 30, 2026 at 7:55 AM Danny McCormick via dev < >> [email protected]> wrote: >> >>> Thanks - could be worth trying, mostly because the cost of trying it is >>> low. >>> >>> I will note that this ends up as a simpler, but also less powerful, >>> version of https://beam.apache.org/documentation/dsls/sql/shell/. It >>> may be worth looking for a way to unify those concepts a little bit more. >>> >>> Any thoughts on the value add of this CLI vs the existing one? >>> >>> > Should a beam console script live in the core apache-beam package, or >>> in a separate package? >>> >>> Putting it in apache-beam seems fine since it won't carry other >>> dependencies AFAIK >>> >>> > Is the scope right for a first step: Python only, Beam YAML only, >>> files plus Kafka and Pub/Sub for auto streaming? >>> >>> I don't understand this question - aren't we supporting all the IOs that >>> Yaml supports? We're also not directly interacting with Python (other than >>> via Yaml), right? >>> >>> Thanks, >>> Danny >>> >>> On Wed, Oct 28, 2026 at 6:41 PM Jignesh Shah via dev < >>> [email protected]> wrote: >>> >>>> >>>> Hi all, >>>> >>>> I'd like feedback on a design for a small beam command-line tool that >>>> makes Beam YAML easier to try: >>>> >>>> Design doc: <Apache Beam CLI Design Proposal >>>> <https://docs.google.com/document/d/1TKEeGGokrcOZhFIdUD-JKgkRfm9a2EYPBkaZyJlnHy4/edit?usp=sharing>, >>>> comments enabled> >>>> >>>> *The idea in two commands* >>>> >>>> pip install 'apache-beam[yaml]' >>>> beam pipe --from users.csv --to out/users.parquet --filter "age >= 21" >>>> beam pipe ... --dry-run > pipeline.yaml # then edit it and: >>>> beam run pipeline.yaml [--runner=...] >>>> >>>> beam run is a thin front end for python -m apache_beam.yaml.main: same >>>> pipelines, same runners and pipeline options. beam pipe covers the >>>> simplest case—moving data between files, Kafka and Pub/Sub topics with an >>>> optional filter and field selection— and prints the equivalent YAML with >>>> --dry-run so users can grow it into a real pipeline. Both run on the >>>> SDK's default runner (Prism, with its usual fallback) unless --runner is >>>> given. No new IO or runner code is involved; it reuses Beam YAML's existing >>>> transforms. >>>> >>>> *Questions where I'd most value input* >>>> >>>> 1. Should a beam console script live in the core apache-beam package, >>>> or in a separate package? >>>> 2. Is the command name pipe right (alternatives in the doc)? >>>> 3. Is the scope right for a first step: Python only, Beam YAML >>>> only, files plus Kafka and Pub/Sub for auto streaming? >>>> 4. For Spark and Flink maintainers: is uber-JAR submission the path >>>> we want to recommend, or make docker a requirement for an easy >>>> experience? >>>> >>>> I'd like to keep comments open for about two weeks before deciding >>>> whether to proceed. If it goes ahead, the plan is a series of small PRs >>>> rather than one large one. >>>> >>>> Disclosure: I used an AI coding assistant to help write the code >>>> prototype and draft the design doc, which I then manually edited. >>>> >>>> Thanks, >>>> >>>> Jignesh >>>> >>>>
