[
https://issues.apache.org/jira/browse/BEAM-1691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15925479#comment-15925479
]
Xu Mingmin commented on BEAM-1691:
----------------------------------
Sure,take an example that I have one job, it may be run on any runner, like
Flink, Spark,, --This is the case when i was testing different runners.
Now my PipelineOptions extends both SparkPipelineOptions and
FlinkPipelineOptions, and package two runners together. *So far it's workable*,
however may break in these scenarios:
1). when SparkPipelineOptions and FlinkPipelineOptions have one same parameter,
it results IllegalArgumentException;
2). suppose SparkPipelineOptions has one option without default value, it's
required when running with FlinkRunner;
Another effect is the package jar is large.
> Dynamic properties supported in PipelineOptions
> -----------------------------------------------
>
> Key: BEAM-1691
> URL: https://issues.apache.org/jira/browse/BEAM-1691
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Reporter: Xu Mingmin
> Assignee: Davor Bonaci
>
> Usually the two lines to create a new Beam pipeline are:
> {code}
> Options options =
> PipelineOptionsFactory.fromArgs(args).withValidation().as(Options.class);
> Pipeline pipeline = Pipeline.create(options);
> {code}
> As each runner has its own PipelineOptions, one piece of code is hardly to
> run on different runners without code change, --as least Options needs to be
> updated.
> Dynamic property could be a choice, similar as
> {code}
> -D property1=value1 -D property2=value2 ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
