Thanks Valentyn for the suggestion but I think a one-shot step is better for a utility.
The closest thing to beam pipe, I can think of is something like beam cp though that doesn't imply transforming during the copy process and hence pipe is better like unix pipe that you use to do transformation ( | tr) It might be better to explain the intended functionality, which I can share from my working branch. https://github.com/jk-shah/beam/blob/beam-cli-preview/.agents/skills/apache-beam-cli/SKILL.md You can try it on your own to see the experience of the complete CLI: https://github.com/jk-shah/beam/blob/beam-cli-preview/sdks/python/cli-preview/PREVIEW_DEMO.md Regards, Jignesh On Tue, Nov 3, 2026 at 3:35 PM Valentyn Tymofieiev <[email protected]> wrote: > > Is the command name pipe right (alternatives in the doc)? > > I would consider naming it `beam create`, or `beam generate` printing the > generated YAML to stdout by default, and allowing `beam run` to accept YAML > from stdin in addition to a file path. > > Sample: > > beam create <arguments> | beam run <pipeline options> > > The two-step approach is still possible: > > beam create <args> > pipeline.yaml > beam run pipeline.yaml [<pipeline options>] > > > Thanks, > Valentyn > > On Tue, Nov 3, 2026 at 12:58 PM Yi Hu via dev <[email protected]> wrote: > >> Thanks for sharing the updates. +1 for the plan (start with a `beam run` >> option and 'beam pipe' option, finally 'beam sql'). For "beam-sql", >> there is also an existing IPython/JupyterLab "magic" supported in Beam: >> >> >> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/interactive/sql/beam_sql_magics.py >> >> similar to that `beam run` invokes `yaml_main`. >> >> Basically one can use `%beam_sql <parameters>` in a CLI fashion, but it >> preserves state (pcollections) across cells in a notebook session. >> >> Regards, >> >> Yi >> >> >> On Mon, Nov 2, 2026 at 10:47 AM Jignesh Shah via dev <[email protected]> >> wrote: >> >>> Over the weekend, I think I prototyped a way to combine both beam pipe' >>> and 'beam-sql' together using 'beam sql' command option (equivalent to the >>> beam-sql.sh script). It does create a dependency to have both Python and >>> Java installed but can be done together as a unified CLI if both languages >>> are installed. Right now my plan is to start with a `beam run` option to >>> set up the configuration in the code, then incorporate the 'beam pipe' >>> option, and finally 'beam sql'. There is also an ease-of-use feature, `beam >>> config`, to simplify common setup across runners, such as selecting a >>> default runner so you do not have to repeat options in every command. This >>> is something I thought was useful because I was repeating the tests to try >>> specific runner configurations, which reduced the required typing. >>> >>> Regards, >>> Jignesh >>> >>> >>> On Fri, Oct 30, 2026 at 1:34 PM Chamikara Jayalath <[email protected]> >>> wrote: >>> >>>> +1 for trying this out as well. I think, ideally, we should have one >>>> command though. The boundary between simple and complex pipelines can vary >>>> based on user expectations, and it's probably best not to make the more >>>> permanent command line APIs depend on that distinction. How about just >>>> using "beam run" for both use-cases but defaulting to the current >>>> definition of "beam pipe" if an argument for the pipeline file, say >>>> "--pipeline" is not provided? >>>> >>>> Thanks, >>>> Cham >>>> >>>> >>>> On Fri, Oct 30, 2026 at 11:52 AM Kenneth Knowles <[email protected]> >>>> wrote: >>>> >>>>> I'm a fan. I like that this offers some of the functionality that >>>>> would otherwise be in `gcloud dataflow` but for all Beam users. I also >>>>> like >>>>> the zero-deps first pipeline. >>>>> >>>>> I'm generally of the belief that now is a time where we should "just >>>>> try stuff" and see what gets used. As long as it doesn't incur a big >>>>> maintenance burden, and has a chance to be useful, I say go for it. (and >>>>> I'm confident the technical details will shake out) >>>>> >>>>> Kenn >>>>> >>>>> On Fri, Oct 30, 2026 at 7:55 AM Danny McCormick via dev < >>>>> [email protected]> wrote: >>>>> >>>>>> Thanks - could be worth trying, mostly because the cost of trying it >>>>>> is low. >>>>>> >>>>>> I will note that this ends up as a simpler, but also less powerful, >>>>>> version of https://beam.apache.org/documentation/dsls/sql/shell/. It >>>>>> may be worth looking for a way to unify those concepts a little bit more. >>>>>> >>>>>> Any thoughts on the value add of this CLI vs the existing one? >>>>>> >>>>>> > Should a beam console script live in the core apache-beam package, >>>>>> or in a separate package? >>>>>> >>>>>> Putting it in apache-beam seems fine since it won't carry other >>>>>> dependencies AFAIK >>>>>> >>>>>> > Is the scope right for a first step: Python only, Beam YAML only, >>>>>> files plus Kafka and Pub/Sub for auto streaming? >>>>>> >>>>>> I don't understand this question - aren't we supporting all the IOs >>>>>> that Yaml supports? We're also not directly interacting with Python >>>>>> (other >>>>>> than via Yaml), right? >>>>>> >>>>>> Thanks, >>>>>> Danny >>>>>> >>>>>> On Wed, Oct 28, 2026 at 6:41 PM Jignesh Shah via dev < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I'd like feedback on a design for a small beam command-line tool >>>>>>> that makes Beam YAML easier to try: >>>>>>> >>>>>>> Design doc: <Apache Beam CLI Design Proposal >>>>>>> <https://docs.google.com/document/d/1TKEeGGokrcOZhFIdUD-JKgkRfm9a2EYPBkaZyJlnHy4/edit?usp=sharing>, >>>>>>> comments enabled> >>>>>>> >>>>>>> *The idea in two commands* >>>>>>> >>>>>>> pip install 'apache-beam[yaml]' >>>>>>> beam pipe --from users.csv --to out/users.parquet --filter "age >= 21" >>>>>>> beam pipe ... --dry-run > pipeline.yaml # then edit it and: >>>>>>> beam run pipeline.yaml [--runner=...] >>>>>>> >>>>>>> beam run is a thin front end for python -m apache_beam.yaml.main: >>>>>>> same pipelines, same runners and pipeline options. beam pipe covers >>>>>>> the simplest case—moving data between files, Kafka and Pub/Sub topics >>>>>>> with >>>>>>> an optional filter and field selection— and prints the equivalent YAML >>>>>>> with >>>>>>> --dry-run so users can grow it into a real pipeline. Both run on >>>>>>> the SDK's default runner (Prism, with its usual fallback) unless >>>>>>> --runner is given. No new IO or runner code is involved; it reuses >>>>>>> Beam YAML's existing transforms. >>>>>>> >>>>>>> *Questions where I'd most value input* >>>>>>> >>>>>>> 1. Should a beam console script live in the core apache-beam package, >>>>>>> or in a separate package? >>>>>>> 2. Is the command name pipe right (alternatives in the doc)? >>>>>>> 3. Is the scope right for a first step: Python only, Beam YAML >>>>>>> only, files plus Kafka and Pub/Sub for auto streaming? >>>>>>> 4. For Spark and Flink maintainers: is uber-JAR submission the >>>>>>> path we want to recommend, or make docker a requirement for an easy >>>>>>> experience? >>>>>>> >>>>>>> I'd like to keep comments open for about two weeks before deciding >>>>>>> whether to proceed. If it goes ahead, the plan is a series of small PRs >>>>>>> rather than one large one. >>>>>>> >>>>>>> Disclosure: I used an AI coding assistant to help write the code >>>>>>> prototype and draft the design doc, which I then manually edited. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Jignesh >>>>>>> >>>>>>>
