[
https://issues.apache.org/jira/browse/BEAM-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943015#comment-16943015
]
Thomas Weise commented on BEAM-8183:
------------------------------------
For context:
[https://lists.apache.org/thread.html/2122928a0a5f678d475ec15af538eb7303f73557870af174b1fdef7e@%3Cdev.beam.apache.org%3E]
Running the same pipeline in different environments with different parameters
is a common need. Virtually everyone has dev/staging/prod or whatever their
environments are and they want to use the same build artifact. That normally
requires some amount of parameterization.
The other use case is bundling multiple pipelines into the same container and
select which to run at launch time.
I was surprised about the question given prior discussion. Even more so
considering that Beam already has the concept of user options. The approach of
generating the jar file currently is equivalent to hard coding all pipeline
options and asking the user to recompile.
Yes, we could generate a new jar file for every option or environment but
please not it bloats the container images (job server is > 100MB). We can also
create separate Docker images, now we are in the GB range.
> Optionally bundle multiple pipelines into a single Flink jar
> ------------------------------------------------------------
>
> Key: BEAM-8183
> URL: https://issues.apache.org/jira/browse/BEAM-8183
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: Major
> Labels: portability-flink
>
> [https://github.com/apache/beam/pull/9331#issuecomment-526734851]
> "With Flink you can bundle multiple entry points into the same jar file and
> specify which one to use with optional flags. It may be desirable to allow
> inclusion of multiple pipelines for this tool also, although that would
> require a different workflow. Absent this option, it becomes quite convoluted
> for users that need the flexibility to choose which pipeline to launch at
> submission time."
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
