[
https://issues.apache.org/jira/browse/BEAM-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948760#comment-16948760
]
Kyle Weaver commented on BEAM-8115:
-----------------------------------
Let's say we want to override some option by providing command line flags to
the jar.
This can be divided into two cases:
# Options that are accessed by the runner: these would be fairly
straightforward to change by simply modifying the options before running the
pipeline.
# Options that are accessed within PTransforms: this is more problematic
because we fully materialize the pipeline before the jar is even created. For
example, let's say we have option --n that we access within PTransform f:
pipeline | Map(lambda x: x + my_options.n)
Here, my_options would be treated as just another global variable, to be
serialized into the Python main session at pipeline creation time. By the time
the jar is run, the main session is already serialized and it would be
unwieldly (if not impossible) to change the main session after that, especially
from a Java program.
(Value providers also don't work here because they too bind values at the time
of pipeline creation, not execution.)
You could probably work around these constraints by having your PTransforms
fetch a parameter from some key-value store at runtime, but of course that
would operate entirely outside of Beam's option mechanisms.
> Overwrite portable Flink application jar pipeline options at runtime
> --------------------------------------------------------------------
>
> Key: BEAM-8115
> URL: https://issues.apache.org/jira/browse/BEAM-8115
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: Major
>
> In the first iteration of portable Flink application jars, all pipeline
> options are set at job creation time and cannot be later modified at runtime.
> There should be a way to pass arguments to the jar to write/overwrite
> pipeline options.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
