[
https://issues.apache.org/jira/browse/BEAM-6942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808216#comment-16808216
]
Valentyn Tymofieiev commented on BEAM-6942:
-------------------------------------------
The issue affects three pipeline options interpreted by Dataflow Runner:
--min_cpu_platform
--enable_streaming_engine
--dataflow_kms_key
Each of these options is supposed to add some experiments, see [1]. However,
this will not happen, when users specify any other experiments on the command
line.
The work around is that users of min_cpu_platform and enable_streaming_engine
pipeline options should also pass experiments to the runner, for example, in
addition to
{noformat}
--min_cpu_platform=Intel\ Haswell --enable_streaming_engine{noformat}
users will have to specify
{noformat}
--experiment=min_cpu_platform=Intel\ Haswell
--experiment=enable_streaming_engine
--experiment=enable_windmill_service{noformat}
This is not a regression, in the sense that this behavior was present in 2.11
release, so I will remove it from 2.12 release blockers.
[1]
[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L356-L388]
> Pipeline options to experiment propagation is not working in Dataflow runner.
> -----------------------------------------------------------------------------
>
> Key: BEAM-6942
> URL: https://issues.apache.org/jira/browse/BEAM-6942
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Valentyn Tymofieiev
> Priority: Critical
> Fix For: 2.12.0
>
>
> Relevant code:
> [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py#L356-L388]
> 3 experiments/options are affected. We need to fix it in 2.12.0
> cc: [~altay] [~apilloud]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
