[
https://issues.apache.org/jira/browse/FLINK-19324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song reassigned FLINK-19324:
------------------------------------
Assignee: Xintong Song
> Map requested/allocated containers with priority on YARN
> --------------------------------------------------------
>
> Key: FLINK-19324
> URL: https://issues.apache.org/jira/browse/FLINK-19324
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Reporter: Xintong Song
> Assignee: Xintong Song
> Priority: Major
>
> In the design doc of FLINK-14106, there was a
> [discussion|https://docs.google.com/document/d/1f8imSus3QwKEUPAldzR8CSMjZ-2a9O17-rn4oeKGtqw/edit?disco=AAAAGPX_tmg]
> on how we map allocated containers with the requested ones on YARN. We
> rejected the design option that uses container priorities for mapping
> containers of different resources, because we do not want to priorities
> different container requests (which is the original purpose for this field).
> As a result, we have to interpret how the requested container request would
> be normalized by Yarn, and map the allocated/requested containers
> accordingly, which is complicated and fragile. See also FLINK-19151.
> Recently in our POC for fine grained resource management, we surprisingly
> discovered that Yarn actually doesn't work with container requests same
> priority and different resources. I do not find this described as an official
> protocol in any Yarn's documents. The issue has been raised in early Yarn
> versions (YARN-314) and has not been fixed util Hadoop 2.9 when
> {{allocationRequestId}} is introduced. In Hadoop 2.8, Yarn scheduler is still
> internally using priority as the key of a container request (see
> [AppSchedulingInfo#updateResourceRequests
> |https://github.com/apache/hadoop/blob/eb818cdc64336ade273a960ba3b9b5a5d0c4d4ec/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java#L341]),
> thus requests same priority and different resources would overwrite each
> other.
> The new discovery suggests that, if we want to support containers with
> different resources on Hadoop 2.8 and earlier versions, we have to give them
> different priorities anyway. Thus, I would suggest to get rid of the
> container normalization simulation and go back to the previously rejected
> priority based design option.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
