+1 for trying this out as well. I think, ideally, we should have one command though. The boundary between simple and complex pipelines can vary based on user expectations, and it's probably best not to make the more permanent command line APIs depend on that distinction. How about just using "beam run" for both use-cases but defaulting to the current definition of "beam pipe" if an argument for the pipeline file, say "--pipeline" is not provided?
Thanks, Cham On Fri, Oct 30, 2026 at 11:52 AM Kenneth Knowles <[email protected]> wrote: > I'm a fan. I like that this offers some of the functionality that would > otherwise be in `gcloud dataflow` but for all Beam users. I also like the > zero-deps first pipeline. > > I'm generally of the belief that now is a time where we should "just try > stuff" and see what gets used. As long as it doesn't incur a big > maintenance burden, and has a chance to be useful, I say go for it. (and > I'm confident the technical details will shake out) > > Kenn > > On Fri, Oct 30, 2026 at 7:55 AM Danny McCormick via dev < > [email protected]> wrote: > >> Thanks - could be worth trying, mostly because the cost of trying it is >> low. >> >> I will note that this ends up as a simpler, but also less powerful, >> version of https://beam.apache.org/documentation/dsls/sql/shell/. It may >> be worth looking for a way to unify those concepts a little bit more. >> >> Any thoughts on the value add of this CLI vs the existing one? >> >> > Should a beam console script live in the core apache-beam package, or >> in a separate package? >> >> Putting it in apache-beam seems fine since it won't carry other >> dependencies AFAIK >> >> > Is the scope right for a first step: Python only, Beam YAML only, files >> plus Kafka and Pub/Sub for auto streaming? >> >> I don't understand this question - aren't we supporting all the IOs that >> Yaml supports? We're also not directly interacting with Python (other than >> via Yaml), right? >> >> Thanks, >> Danny >> >> On Wed, Oct 28, 2026 at 6:41 PM Jignesh Shah via dev <[email protected]> >> wrote: >> >>> >>> Hi all, >>> >>> I'd like feedback on a design for a small beam command-line tool that >>> makes Beam YAML easier to try: >>> >>> Design doc: <Apache Beam CLI Design Proposal >>> <https://docs.google.com/document/d/1TKEeGGokrcOZhFIdUD-JKgkRfm9a2EYPBkaZyJlnHy4/edit?usp=sharing>, >>> comments enabled> >>> >>> *The idea in two commands* >>> >>> pip install 'apache-beam[yaml]' >>> beam pipe --from users.csv --to out/users.parquet --filter "age >= 21" >>> beam pipe ... --dry-run > pipeline.yaml # then edit it and: >>> beam run pipeline.yaml [--runner=...] >>> >>> beam run is a thin front end for python -m apache_beam.yaml.main: same >>> pipelines, same runners and pipeline options. beam pipe covers the >>> simplest case—moving data between files, Kafka and Pub/Sub topics with an >>> optional filter and field selection— and prints the equivalent YAML with >>> --dry-run so users can grow it into a real pipeline. Both run on the >>> SDK's default runner (Prism, with its usual fallback) unless --runner is >>> given. No new IO or runner code is involved; it reuses Beam YAML's existing >>> transforms. >>> >>> *Questions where I'd most value input* >>> >>> 1. Should a beam console script live in the core apache-beam package, >>> or in a separate package? >>> 2. Is the command name pipe right (alternatives in the doc)? >>> 3. Is the scope right for a first step: Python only, Beam YAML only, >>> files plus Kafka and Pub/Sub for auto streaming? >>> 4. For Spark and Flink maintainers: is uber-JAR submission the path >>> we want to recommend, or make docker a requirement for an easy >>> experience? >>> >>> I'd like to keep comments open for about two weeks before deciding >>> whether to proceed. If it goes ahead, the plan is a series of small PRs >>> rather than one large one. >>> >>> Disclosure: I used an AI coding assistant to help write the code >>> prototype and draft the design doc, which I then manually edited. >>> >>> Thanks, >>> >>> Jignesh >>> >>>
