Hi all, I'd like feedback on a design for a small beam command-line tool that makes Beam YAML easier to try:
Design doc: <Apache Beam CLI Design Proposal <https://docs.google.com/document/d/1TKEeGGokrcOZhFIdUD-JKgkRfm9a2EYPBkaZyJlnHy4/edit?usp=sharing>, comments enabled> *The idea in two commands* pip install 'apache-beam[yaml]' beam pipe --from users.csv --to out/users.parquet --filter "age >= 21" beam pipe ... --dry-run > pipeline.yaml # then edit it and: beam run pipeline.yaml [--runner=...] beam run is a thin front end for python -m apache_beam.yaml.main: same pipelines, same runners and pipeline options. beam pipe covers the simplest case—moving data between files, Kafka and Pub/Sub topics with an optional filter and field selection— and prints the equivalent YAML with --dry-run so users can grow it into a real pipeline. Both run on the SDK's default runner (Prism, with its usual fallback) unless --runner is given. No new IO or runner code is involved; it reuses Beam YAML's existing transforms. *Questions where I'd most value input* 1. Should a beam console script live in the core apache-beam package, or in a separate package? 2. Is the command name pipe right (alternatives in the doc)? 3. Is the scope right for a first step: Python only, Beam YAML only, files plus Kafka and Pub/Sub for auto streaming? 4. For Spark and Flink maintainers: is uber-JAR submission the path we want to recommend, or make docker a requirement for an easy experience? I'd like to keep comments open for about two weeks before deciding whether to proceed. If it goes ahead, the plan is a series of small PRs rather than one large one. Disclosure: I used an AI coding assistant to help write the code prototype and draft the design doc, which I then manually edited. Thanks, Jignesh
