Hello! What Beam aims to provide is keeping the same code of the pipeline itself when you switch from one runner to another. But Options is runner-specific.
When you change runners, you still need to rebuild the project with the correct runner and runner options. For example: https://beam.apache.org/releases/javadoc/current/org/apache/beam/runners/flink/FlinkPipelineOptions.html https://beam.apache.org/releases/javadoc/current/org/apache/beam/runners/dataflow/options/DataflowPipelineOptions.html In my Beam projects I was managing that via maven profiles: - pipeline code is packed into its own module, does not channge when I switch runners, and is always included into the final artifact - pipeline options and Main have very little code (only calls generic pipeline code), are runner-specific, and I compile them based on the profile selected -P flinkRunner / -P dataflowRunner / etc On Sunday, 7 May 2023, Mikhail Khludnev <m...@apache.org> wrote: > Hello, > I'm starting to learn Beam. I took Java SDK and play with WordCount/Starter. > I'm able to run pipeline in embedded Flink via --runner=FlinkRunner > INFO: Starting Flink Mini Cluster > However passing --flinkMaster=localhost:8081 doesn't really work due to > java.lang.IllegalArgumentException: Class interface WordCountOptions missing a property named 'flinkMaster'. > It somewhat reasonable because > interface WordCountOptions extends PipelineOptions { > I can call withoutStrictParsing() that just bypasses check, but doesn't pass flinkMaster argument to FlinkRunner and I've got Flink Mini Cluster again. > Another way is to extends my custom options from FlinkPipelineOptions it makes it work on remote cluster. Fine. > Here I'm missing something, it seems like I can't run this pipeline by another runner, let's say Dataflow without changing the code. > How can I build a runner agnostic pipeline? > Sorry for asking a dummy question. > > -- > Sincerely yours > Mikhail Khludnev > -- Best Regards, Pavel Solomin Tel: +351 962 950 692 | Skype: pavel_solomin | Linkedin <https://www.linkedin.com/in/pavelsolomin>
