[
https://issues.apache.org/jira/browse/FLINK-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Hogan closed FLINK-1703.
-----------------------------
Resolution: Duplicate
> option to balance load according to CPU available rather than slots available
> -----------------------------------------------------------------------------
>
> Key: FLINK-1703
> URL: https://issues.apache.org/jira/browse/FLINK-1703
> Project: Flink
> Issue Type: Improvement
> Components: Streaming
> Reporter: Emmanuel Leroy
> Labels: balancer, load
>
> To be able to accommodate for loading and running several different
> topologies on a cluster, one may want to assign more slots per node than
> there are CPUs available (many slots per CPU).
>
> For example, with 3 nodes with 3 CPUs each, defining 3 slots per node (9
> slots total) limits the number of task that can be loaded at any one time.
> => Increasing the number of slots to 3 per CPU, gives 27 slots, allowing to
> load more topologies.
> In this configuration (3 nodes, 3 CPUs, 27 slots), if a job specifies a
> parallelism of 9, 9 slots out of 27 are used, but the distribution strategy
> gives preference to slots on the same host as the stream source to limit
> network IOs: as a consequence, the first job using 9 slots has the 9 slots
> allocated on a single host (the one on the same host as the stream source),
> even though this host only has 3 CPUs available.
> These 9 slots are therefore sharing the same 3 CPUs, while the other 2 nodes
> (6 CPUs) stay idle.
> The distribution strategy should take into account the number of CPUs
> available along with the number of slots rather than rely only on the slot
> availability.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
