[
https://issues.apache.org/jira/browse/BEAM-3089?focusedWorklogId=145308&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145308
]
ASF GitHub Bot logged work on BEAM-3089:
----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Sep/18 14:42
Start Date: 18/Sep/18 14:42
Worklog Time Spent: 10m
Work Description: mxm opened a new pull request #6426: [BEAM-3089] Fix
default values in FlinkPipelineOptions / Add tests
URL: https://github.com/apache/beam/pull/6426
- [BEAM-3089] Use Flink cluster parallelism if no parallelism provided
The Runner always defaulted to 1, even if the Flink cluster had a default
parallelism set. With this patch, when no parallelism has been provided, the
cluster default will be used.
This was working before, it is a regression of
cdd2544.
- [BEAM-3089] Add test for FlinkExecutionEnvironments
- Revert default checkpointing mode to EXACTLY_ONCE
- Use default for checkpoint timeout
- [BEAM-3089] Test default values of FlinkPipelineOptions
- Set default master url to [auto]
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| --- | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 145308)
Time Spent: 1.5h (was: 1h 20m)
> Issue with setting the parallelism at client level using Flink runner
> ---------------------------------------------------------------------
>
> Key: BEAM-3089
> URL: https://issues.apache.org/jira/browse/BEAM-3089
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Affects Versions: 2.0.0
> Environment: I am using Flink 1.2.1 running on Docker, with Task
> Managers distributed across different VMs as part of a Docker Swarm.
> Reporter: Thalita Vergilio
> Assignee: Grzegorz KoĊakowski
> Priority: Major
> Labels: docker, flink, parallel-deployment
> Fix For: 2.8.0
>
> Attachments: flink-ui-parallelism.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When uploading an Apache Beam application using the Flink Web UI, the
> parallelism set at job submission doesn't get picked up. The same happens
> when submitting a job using the Flink CLI.
> In both cases, the parallelism ends up defaulting to 1.
> When I set the parallelism programmatically within the Apache Beam code, it
> works: {{flinkPipelineOptions.setParallelism(4);}}
> I suspect the root of the problem may be in the
> org.apache.beam.runners.flink.DefaultParallelismFactory class, as it checks
> for Flink's GlobalConfiguration, which may not pick up runtime values passed
> to Flink, then defaults to 1 if it doesn't find anything.
> Any ideas on how this could be fixed or worked around? I need to be able to
> change the parallelism dynamically, so the programmatic approach won't really
> work for me, nor will setting the Flink configuration at system level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
