[jira] [Updated] (BEAM-3089) Issue with setting the parallelism at client level using Flink runner
[ https://issues.apache.org/jira/browse/BEAM-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maximilian Michels updated BEAM-3089: - Fix Version/s: 2.8.0 > Issue with setting the parallelism at client level using Flink runner > - > > Key: BEAM-3089 > URL: https://issues.apache.org/jira/browse/BEAM-3089 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.0.0 > Environment: I am using Flink 1.2.1 running on Docker, with Task > Managers distributed across different VMs as part of a Docker Swarm. >Reporter: Thalita Vergilio >Assignee: Grzegorz Kołakowski >Priority: Major > Labels: docker, flink, parallel-deployment > Fix For: 2.8.0 > > Attachments: flink-ui-parallelism.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > When uploading an Apache Beam application using the Flink Web UI, the > parallelism set at job submission doesn't get picked up. The same happens > when submitting a job using the Flink CLI. > In both cases, the parallelism ends up defaulting to 1. > When I set the parallelism programmatically within the Apache Beam code, it > works: {{flinkPipelineOptions.setParallelism(4);}} > I suspect the root of the problem may be in the > org.apache.beam.runners.flink.DefaultParallelismFactory class, as it checks > for Flink's GlobalConfiguration, which may not pick up runtime values passed > to Flink, then defaults to 1 if it doesn't find anything. > Any ideas on how this could be fixed or worked around? I need to be able to > change the parallelism dynamically, so the programmatic approach won't really > work for me, nor will setting the Flink configuration at system level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3089) Issue with setting the parallelism at client level using Flink runner
[ https://issues.apache.org/jira/browse/BEAM-3089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grzegorz Kołakowski updated BEAM-3089: -- Attachment: flink-ui-parallelism.png > Issue with setting the parallelism at client level using Flink runner > - > > Key: BEAM-3089 > URL: https://issues.apache.org/jira/browse/BEAM-3089 > Project: Beam > Issue Type: Bug > Components: runner-flink >Affects Versions: 2.0.0 > Environment: I am using Flink 1.2.1 running on Docker, with Task > Managers distributed across different VMs as part of a Docker Swarm. >Reporter: Thalita Vergilio >Assignee: Aljoscha Krettek >Priority: Major > Labels: docker, flink, parallel-deployment > Attachments: flink-ui-parallelism.png > > > When uploading an Apache Beam application using the Flink Web UI, the > parallelism set at job submission doesn't get picked up. The same happens > when submitting a job using the Flink CLI. > In both cases, the parallelism ends up defaulting to 1. > When I set the parallelism programmatically within the Apache Beam code, it > works: {{flinkPipelineOptions.setParallelism(4);}} > I suspect the root of the problem may be in the > org.apache.beam.runners.flink.DefaultParallelismFactory class, as it checks > for Flink's GlobalConfiguration, which may not pick up runtime values passed > to Flink, then defaults to 1 if it doesn't find anything. > Any ideas on how this could be fixed or worked around? I need to be able to > change the parallelism dynamically, so the programmatic approach won't really > work for me, nor will setting the Flink configuration at system level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)