Tony Xintong Song created FLINK-11166:
-----------------------------------------
Summary: Slot Placement Constraint
Key: FLINK-11166
URL: https://issues.apache.org/jira/browse/FLINK-11166
Project: Flink
Issue Type: New Feature
Components: ResourceManager
Reporter: Tony Xintong Song
In many cases, users may want Flink to schedule their job tasks following
certain locality preferences. E.g., colocating upstream/downstream tasks to
reduce data transmission costs, dispersing tasks of certain pattern (e.g., I/O
intensive) to avoid resource competitions, running tasks in exclusive
TaskExecutor-s for task level resource consumption measurements, etc.
Currently, there are two ways in Flink to specify such locality preferences:
specifying preferred locations in the slot request, or setting slot sharing
group for the task. In both ways the preferences are specified when requesting
slots from the SlotPool and can affect how tasks are placed among the slots
allocated to the JobMaster.
However, there is no guarantee that such preferences can always be satisfied,
especially when slots are customized with different resource profiles for
different kinds of tasks. E.g., in cases where two tasks A and B are preferred
to be scheduled onto a same TaskExecutor, it is possible that none of the slots
customized for A offered to the JobMaster are collocated with slots customized
for B.
To better support locality preferences with various slot resource
specifications, it is necessary to allow JobMaster-s to request slots subjected
to certain placement constraints from the ResourceManager.
In addition, most underlying frameworks Flink runs on (Yarn, Kubernetes, Mesos)
already have individual supports for container level placement constraints. It
is a great opportunity for Flink to leverage such underlying supports and
enable scheduling tasks with rich locality preferences.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
