Emmanuel Leroy created FLINK-1703:
-------------------------------------
Summary: option to balance load according to CPU available rather
than slots available
Key: FLINK-1703
URL: https://issues.apache.org/jira/browse/FLINK-1703
Project: Flink
Issue Type: Improvement
Components: Streaming
Reporter: Emmanuel Leroy
To be able to accommodate for loading and running several different topologies
on a cluster, one may want to assign more slots per node than there are CPUs
available (many slots per CPU).
For example, with 3 nodes with 3 CPUs each, defining 3 slots per node (9 slots
total) limits the number of task that can be loaded at any one time.
=> Increasing the number of slots to 3 per CPU, gives 27 slots, allowing to
load more topologies.
In this configuration (3 nodes, 3 CPUs, 27 slots), if a job specifies a
parallelism of 9, 9 slots out of 27 are used, but the distribution strategy
gives preference to slots on the same host as the stream source to limit
network IOs: as a consequence, the first job using 9 slots has the 9 slots
allocated on a single host (the one on the same host as the stream source),
even though this host only has 3 CPUs available.
These 9 slots are therefore sharing the same 3 CPUs, while the other 2 nodes (6
CPUs) stay idle.
The distribution strategy should take into account the number of CPUs available
along with the number of slots rather than rely only on the slot availability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
