Thanks - could be worth trying, mostly because the cost of trying it is low.
I will note that this ends up as a simpler, but also less powerful, version of https://beam.apache.org/documentation/dsls/sql/shell/. It may be worth looking for a way to unify those concepts a little bit more. Any thoughts on the value add of this CLI vs the existing one? > Should a beam console script live in the core apache-beam package, or in a separate package? Putting it in apache-beam seems fine since it won't carry other dependencies AFAIK > Is the scope right for a first step: Python only, Beam YAML only, files plus Kafka and Pub/Sub for auto streaming? I don't understand this question - aren't we supporting all the IOs that Yaml supports? We're also not directly interacting with Python (other than via Yaml), right? Thanks, Danny On Wed, Oct 28, 2026 at 6:41 PM Jignesh Shah via dev <[email protected]> wrote: > > Hi all, > > I'd like feedback on a design for a small beam command-line tool that > makes Beam YAML easier to try: > > Design doc: <Apache Beam CLI Design Proposal > <https://docs.google.com/document/d/1TKEeGGokrcOZhFIdUD-JKgkRfm9a2EYPBkaZyJlnHy4/edit?usp=sharing>, > comments enabled> > > *The idea in two commands* > > pip install 'apache-beam[yaml]' > beam pipe --from users.csv --to out/users.parquet --filter "age >= 21" > beam pipe ... --dry-run > pipeline.yaml # then edit it and: > beam run pipeline.yaml [--runner=...] > > beam run is a thin front end for python -m apache_beam.yaml.main: same > pipelines, same runners and pipeline options. beam pipe covers the > simplest case—moving data between files, Kafka and Pub/Sub topics with an > optional filter and field selection— and prints the equivalent YAML with > --dry-run so users can grow it into a real pipeline. Both run on the > SDK's default runner (Prism, with its usual fallback) unless --runner is > given. No new IO or runner code is involved; it reuses Beam YAML's existing > transforms. > > *Questions where I'd most value input* > > 1. Should a beam console script live in the core apache-beam package, > or in a separate package? > 2. Is the command name pipe right (alternatives in the doc)? > 3. Is the scope right for a first step: Python only, Beam YAML only, > files plus Kafka and Pub/Sub for auto streaming? > 4. For Spark and Flink maintainers: is uber-JAR submission the path we > want to recommend, or make docker a requirement for an easy experience? > > I'd like to keep comments open for about two weeks before deciding whether > to proceed. If it goes ahead, the plan is a series of small PRs rather than > one large one. > > Disclosure: I used an AI coding assistant to help write the code prototype > and draft the design doc, which I then manually edited. > > Thanks, > > Jignesh > >
