> Is the command name pipe right (alternatives in the doc)? I would consider naming it `beam create`, or `beam generate` printing the generated YAML to stdout by default, and allowing `beam run` to accept YAML from stdin in addition to a file path.
Sample: beam create <arguments> | beam run <pipeline options> The two-step approach is still possible: beam create <args> > pipeline.yaml beam run pipeline.yaml [<pipeline options>] Thanks, Valentyn On Tue, Nov 3, 2026 at 12:58 PM Yi Hu via dev <[email protected]> wrote: > Thanks for sharing the updates. +1 for the plan (start with a `beam run` > option and 'beam pipe' option, finally 'beam sql'). For "beam-sql", there > is also an existing IPython/JupyterLab "magic" supported in Beam: > > > https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/sql/beam_sql_magics.py > > similar to that `beam run` invokes `yaml_main`. > > Basically one can use `%beam_sql <parameters>` in a CLI fashion, but it > preserves state (pcollections) across cells in a notebook session. > > Regards, > > Yi > > > On Mon, Nov 2, 2026 at 10:47 AM Jignesh Shah via dev <[email protected]> > wrote: > >> Over the weekend, I think I prototyped a way to combine both beam pipe' >> and 'beam-sql' together using 'beam sql' command option (equivalent to the >> beam-sql.sh script). It does create a dependency to have both Python and >> Java installed but can be done together as a unified CLI if both languages >> are installed. Right now my plan is to start with a `beam run` option to >> set up the configuration in the code, then incorporate the 'beam pipe' >> option, and finally 'beam sql'. There is also an ease-of-use feature, `beam >> config`, to simplify common setup across runners, such as selecting a >> default runner so you do not have to repeat options in every command. This >> is something I thought was useful because I was repeating the tests to try >> specific runner configurations, which reduced the required typing. >> >> Regards, >> Jignesh >> >> >> On Fri, Oct 30, 2026 at 1:34 PM Chamikara Jayalath <[email protected]> >> wrote: >> >>> +1 for trying this out as well. I think, ideally, we should have one >>> command though. The boundary between simple and complex pipelines can vary >>> based on user expectations, and it's probably best not to make the more >>> permanent command line APIs depend on that distinction. How about just >>> using "beam run" for both use-cases but defaulting to the current >>> definition of "beam pipe" if an argument for the pipeline file, say >>> "--pipeline" is not provided? >>> >>> Thanks, >>> Cham >>> >>> >>> On Fri, Oct 30, 2026 at 11:52 AM Kenneth Knowles <[email protected]> >>> wrote: >>> >>>> I'm a fan. I like that this offers some of the functionality that would >>>> otherwise be in `gcloud dataflow` but for all Beam users. I also like the >>>> zero-deps first pipeline. >>>> >>>> I'm generally of the belief that now is a time where we should "just >>>> try stuff" and see what gets used. As long as it doesn't incur a big >>>> maintenance burden, and has a chance to be useful, I say go for it. (and >>>> I'm confident the technical details will shake out) >>>> >>>> Kenn >>>> >>>> On Fri, Oct 30, 2026 at 7:55 AM Danny McCormick via dev < >>>> [email protected]> wrote: >>>> >>>>> Thanks - could be worth trying, mostly because the cost of trying it >>>>> is low. >>>>> >>>>> I will note that this ends up as a simpler, but also less powerful, >>>>> version of https://beam.apache.org/documentation/dsls/sql/shell/. It >>>>> may be worth looking for a way to unify those concepts a little bit more. >>>>> >>>>> Any thoughts on the value add of this CLI vs the existing one? >>>>> >>>>> > Should a beam console script live in the core apache-beam package, >>>>> or in a separate package? >>>>> >>>>> Putting it in apache-beam seems fine since it won't carry other >>>>> dependencies AFAIK >>>>> >>>>> > Is the scope right for a first step: Python only, Beam YAML only, >>>>> files plus Kafka and Pub/Sub for auto streaming? >>>>> >>>>> I don't understand this question - aren't we supporting all the IOs >>>>> that Yaml supports? We're also not directly interacting with Python (other >>>>> than via Yaml), right? >>>>> >>>>> Thanks, >>>>> Danny >>>>> >>>>> On Wed, Oct 28, 2026 at 6:41 PM Jignesh Shah via dev < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I'd like feedback on a design for a small beam command-line tool >>>>>> that makes Beam YAML easier to try: >>>>>> >>>>>> Design doc: <Apache Beam CLI Design Proposal >>>>>> <https://docs.google.com/document/d/1TKEeGGokrcOZhFIdUD-JKgkRfm9a2EYPBkaZyJlnHy4/edit?usp=sharing>, >>>>>> comments enabled> >>>>>> >>>>>> *The idea in two commands* >>>>>> >>>>>> pip install 'apache-beam[yaml]' >>>>>> beam pipe --from users.csv --to out/users.parquet --filter "age >= 21" >>>>>> beam pipe ... --dry-run > pipeline.yaml # then edit it and: >>>>>> beam run pipeline.yaml [--runner=...] >>>>>> >>>>>> beam run is a thin front end for python -m apache_beam.yaml.main: >>>>>> same pipelines, same runners and pipeline options. beam pipe covers >>>>>> the simplest case—moving data between files, Kafka and Pub/Sub topics >>>>>> with >>>>>> an optional filter and field selection— and prints the equivalent YAML >>>>>> with >>>>>> --dry-run so users can grow it into a real pipeline. Both run on the >>>>>> SDK's default runner (Prism, with its usual fallback) unless --runner is >>>>>> given. No new IO or runner code is involved; it reuses Beam YAML's >>>>>> existing >>>>>> transforms. >>>>>> >>>>>> *Questions where I'd most value input* >>>>>> >>>>>> 1. Should a beam console script live in the core apache-beam package, >>>>>> or in a separate package? >>>>>> 2. Is the command name pipe right (alternatives in the doc)? >>>>>> 3. Is the scope right for a first step: Python only, Beam YAML >>>>>> only, files plus Kafka and Pub/Sub for auto streaming? >>>>>> 4. For Spark and Flink maintainers: is uber-JAR submission the >>>>>> path we want to recommend, or make docker a requirement for an easy >>>>>> experience? >>>>>> >>>>>> I'd like to keep comments open for about two weeks before deciding >>>>>> whether to proceed. If it goes ahead, the plan is a series of small PRs >>>>>> rather than one large one. >>>>>> >>>>>> Disclosure: I used an AI coding assistant to help write the code >>>>>> prototype and draft the design doc, which I then manually edited. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Jignesh >>>>>> >>>>>>
