[
https://issues.apache.org/jira/browse/YARN-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942731#comment-13942731
]
Rohith commented on YARN-1198:
------------------------------
bq. It's kind of related to "New node is added/removed from the cluster" above
In YARN-1680, from yarn cluster perspective, number of NodeManager remain
same. Applicationmaster marked 1 nodemanager as blacklisted and update to RM.
Further RM does not assign any containers on blacklisted nodes. But headroom
sent to applicationmaster(availableResource) is per cluster level.
Say 4 NM's(NM1,NM2,NM3,NM4) in cluster with 8GB each.
NM1,NM2,NM3 and NM4 running task occuping 27GB of whole cluster.*5GB free in
NM4*
Now Headroom=5GB(RM calculated and sent to applicationmaster)
After *NM4 is blacklisted* by applicationmaster,still *headroom=5GB*(RM
calculates headroom including NM4). This is wrong value receiving by
applicationmaster!!!.
> Capacity Scheduler headroom calculation does not work as expected
> -----------------------------------------------------------------
>
> Key: YARN-1198
> URL: https://issues.apache.org/jira/browse/YARN-1198
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Omkar Vinit Joshi
> Assignee: Omkar Vinit Joshi
>
> Today headroom calculation (for the app) takes place only when
> * New node is added/removed from the cluster
> * New container is getting assigned to the application.
> However there are potentially lot of situations which are not considered for
> this calculation
> * If a container finishes then headroom for that application will change and
> should be notified to the AM accordingly.
> * If a single user has submitted multiple applications (app1 and app2) to the
> same queue then
> ** If app1's container finishes then not only app1's but also app2's AM
> should be notified about the change in headroom.
> ** Similarly if a container is assigned to any applications app1/app2 then
> both AM should be notified about their headroom.
> ** To simplify the whole communication process it is ideal to keep headroom
> per User per LeafQueue so that everyone gets the same picture (apps belonging
> to same user and submitted in same queue).
> * If a new user submits an application to the queue then all applications
> submitted by all users in that queue should be notified of the headroom
> change.
> * Also today headroom is an absolute number ( I think it should be normalized
> but then this is going to be not backward compatible..)
> * Also when admin user refreshes queue headroom has to be updated.
> These all are the potential bugs in headroom calculations
--
This message was sent by Atlassian JIRA
(v6.2#6252)
