I'm a fan. I like that this offers some of the functionality that would otherwise be in `gcloud dataflow` but for all Beam users. I also like the zero-deps first pipeline.
I'm generally of the belief that now is a time where we should "just try stuff" and see what gets used. As long as it doesn't incur a big maintenance burden, and has a chance to be useful, I say go for it. (and I'm confident the technical details will shake out) Kenn On Fri, Oct 30, 2026 at 7:55 AM Danny McCormick via dev <[email protected]> wrote: > Thanks - could be worth trying, mostly because the cost of trying it is > low. > > I will note that this ends up as a simpler, but also less powerful, > version of https://beam.apache.org/documentation/dsls/sql/shell/. It may > be worth looking for a way to unify those concepts a little bit more. > > Any thoughts on the value add of this CLI vs the existing one? > > > Should a beam console script live in the core apache-beam package, or in > a separate package? > > Putting it in apache-beam seems fine since it won't carry other > dependencies AFAIK > > > Is the scope right for a first step: Python only, Beam YAML only, files > plus Kafka and Pub/Sub for auto streaming? > > I don't understand this question - aren't we supporting all the IOs that > Yaml supports? We're also not directly interacting with Python (other than > via Yaml), right? > > Thanks, > Danny > > On Wed, Oct 28, 2026 at 6:41 PM Jignesh Shah via dev <[email protected]> > wrote: > >> >> Hi all, >> >> I'd like feedback on a design for a small beam command-line tool that >> makes Beam YAML easier to try: >> >> Design doc: <Apache Beam CLI Design Proposal >> <https://docs.google.com/document/d/1TKEeGGokrcOZhFIdUD-JKgkRfm9a2EYPBkaZyJlnHy4/edit?usp=sharing>, >> comments enabled> >> >> *The idea in two commands* >> >> pip install 'apache-beam[yaml]' >> beam pipe --from users.csv --to out/users.parquet --filter "age >= 21" >> beam pipe ... --dry-run > pipeline.yaml # then edit it and: >> beam run pipeline.yaml [--runner=...] >> >> beam run is a thin front end for python -m apache_beam.yaml.main: same >> pipelines, same runners and pipeline options. beam pipe covers the >> simplest case—moving data between files, Kafka and Pub/Sub topics with an >> optional filter and field selection— and prints the equivalent YAML with >> --dry-run so users can grow it into a real pipeline. Both run on the >> SDK's default runner (Prism, with its usual fallback) unless --runner is >> given. No new IO or runner code is involved; it reuses Beam YAML's existing >> transforms. >> >> *Questions where I'd most value input* >> >> 1. Should a beam console script live in the core apache-beam package, >> or in a separate package? >> 2. Is the command name pipe right (alternatives in the doc)? >> 3. Is the scope right for a first step: Python only, Beam YAML only, >> files plus Kafka and Pub/Sub for auto streaming? >> 4. For Spark and Flink maintainers: is uber-JAR submission the path >> we want to recommend, or make docker a requirement for an easy experience? >> >> I'd like to keep comments open for about two weeks before deciding >> whether to proceed. If it goes ahead, the plan is a series of small PRs >> rather than one large one. >> >> Disclosure: I used an AI coding assistant to help write the code >> prototype and draft the design doc, which I then manually edited. >> >> Thanks, >> >> Jignesh >> >>
