[
https://issues.apache.org/jira/browse/BEAM-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943201#comment-16943201
]
Ankur Goenka commented on BEAM-8183:
------------------------------------
I agree, changing a bit of configuration in the proto will serve a lot of use
cases. A few can be the input/output data file etc.
{quote}
You are correct that the Python entry point / driver program would need to be
(re)executed for a fully generic solution. But that's not necessary for the
majority of use cases. Those are artifact + configuration. If there is a way to
parameterize configuration values in the proto, we can address that majority of
use cases with a single job jar artifact.
{quote}
Will
[value_provider|[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/options/value_provider.py]]
help in this case? Dataflow templates use this.
Also, we can enhance the driver class to swap the actual option values in the
options proto to parameters provided at the submission time.
{quote}
But beyond that we also have (in our infrastructure) the use case of multiple
entry points that the user can pick at submit time.
{quote}
Thats a valid usecase. I can't imagine a good way to model it in beam as all
the beam notions are build considering a single pipeline at a time. Will a
shell script capable of merging merging the jars for different pipeline.
I think a pipeline docker can resolve a lot of these issues as it will be
capable of running the submission code in a consistent manner based on the
arguments provided.
> Optionally bundle multiple pipelines into a single Flink jar
> ------------------------------------------------------------
>
> Key: BEAM-8183
> URL: https://issues.apache.org/jira/browse/BEAM-8183
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: Major
> Labels: portability-flink
>
> [https://github.com/apache/beam/pull/9331#issuecomment-526734851]
> "With Flink you can bundle multiple entry points into the same jar file and
> specify which one to use with optional flags. It may be desirable to allow
> inclusion of multiple pipelines for this tool also, although that would
> require a different workflow. Absent this option, it becomes quite convoluted
> for users that need the flexibility to choose which pipeline to launch at
> submission time."
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
