[
https://issues.apache.org/jira/browse/FLINK-1709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ufuk Celebi updated FLINK-1709:
-------------------------------
Summary: Add support for programs with slot count exceeding parallelism
(was: Add support for programs with higher-than-slot parallelism)
> Add support for programs with slot count exceeding parallelism
> --------------------------------------------------------------
>
> Key: FLINK-1709
> URL: https://issues.apache.org/jira/browse/FLINK-1709
> Project: Flink
> Issue Type: Improvement
> Affects Versions: master
> Reporter: Ufuk Celebi
>
> Currently, we can't run programs with higher parallelism than available slots.
> For example, if we currently have a map-reduce program and 4 task slots
> configured (e.g. 2 task managers with 2 slots per task manager), the map and
> reduce tasks will be scheduled with pipelined results and the same
> parallelism in shared slots. Setting the parallelism to more than available
> slots will result in a NoResourcesAvailableException.
> As a first step to support these kinds of programs, we can add initial
> support for this when running in batch mode (after
> https://github.com/apache/flink/pull/471 is merged).
> This is easier than the original pipelined scenario, because the map tasks
> can be deployed after each other to produce the blocking result. The blocking
> result can then be consumed after all map tasks produced their result. The
> mechanism in #471 to deploy result receivers can be used for this and should
> not need any modifications.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
