[
https://issues.apache.org/jira/browse/BEAM-8183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943170#comment-16943170
]
Thomas Weise commented on BEAM-8183:
------------------------------------
{quote}The major issue to me seems to be that we need to execute pipeline
construction code which is environment dependent. To generate new pipelines for
an environment, we need to execute the pipeline submission code in that
environment. And this is where I see a problem. Python pipelines have to
execute user code in python using python sdk to construct the pipeline.
{quote}
You are correct that the Python entry point / driver program would need to be
(re)executed for a fully generic solution. But that's not necessary for the
majority of use cases. Those are artifact + configuration. If there is a way to
parameterize configuration values in the proto, we can address that majority of
use cases with a single job jar artifact.
My fallback for the exception path would be to generate multiple protos into a
single jar, which is why I'm interested in this capability. So that jar would
contain "mypipeline_staging" and "mypipeline_production" and the deployment
would select the pipeline via its configuration (parameter to the Flink entry
point). Similar would work for Spark.
But beyond that we also have (in our infrastructure) the use case of multiple
entry points that the user can pick at submit time.
> Optionally bundle multiple pipelines into a single Flink jar
> ------------------------------------------------------------
>
> Key: BEAM-8183
> URL: https://issues.apache.org/jira/browse/BEAM-8183
> Project: Beam
> Issue Type: New Feature
> Components: runner-flink
> Reporter: Kyle Weaver
> Assignee: Kyle Weaver
> Priority: Major
> Labels: portability-flink
>
> [https://github.com/apache/beam/pull/9331#issuecomment-526734851]
> "With Flink you can bundle multiple entry points into the same jar file and
> specify which one to use with optional flags. It may be desirable to allow
> inclusion of multiple pipelines for this tool also, although that would
> require a different workflow. Absent this option, it becomes quite convoluted
> for users that need the flexibility to choose which pipeline to launch at
> submission time."
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
