Wangda Tan commented on YARN-1198:

I agree with [~jlowe], [~airbots] and [~cwelch], used resource should be 
considered into headroom (which is YANR-2008). And apparently, application 
master can ask more than that number to get more resource possibly. 

I completely agree with what Jason mentioned, ignore headroom will not cause 
more problem except application itself. What I originally want to say is when 
putting headroom and gang scheduling together, it will cause deadlock problem 
and should be solved in scheduler side. But it seems kind of off-topic, let's 
ignore it here. 

Also, as Chen mentioned, we don't need consider preemption when computing 
headroom. And besides, when resource will be preempted from an app, the AM will 
receive messages about preemption requests, it should handle itself.


> Capacity Scheduler headroom calculation does not work as expected
> -----------------------------------------------------------------
>                 Key: YARN-1198
>                 URL: https://issues.apache.org/jira/browse/YARN-1198
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Omkar Vinit Joshi
>            Assignee: Omkar Vinit Joshi
>         Attachments: YARN-1198.1.patch
> Today headroom calculation (for the app) takes place only when
> * New node is added/removed from the cluster
> * New container is getting assigned to the application.
> However there are potentially lot of situations which are not considered for 
> this calculation
> * If a container finishes then headroom for that application will change and 
> should be notified to the AM accordingly.
> * If a single user has submitted multiple applications (app1 and app2) to the 
> same queue then
> ** If app1's container finishes then not only app1's but also app2's AM 
> should be notified about the change in headroom.
> ** Similarly if a container is assigned to any applications app1/app2 then 
> both AM should be notified about their headroom.
> ** To simplify the whole communication process it is ideal to keep headroom 
> per User per LeafQueue so that everyone gets the same picture (apps belonging 
> to same user and submitted in same queue).
> * If a new user submits an application to the queue then all applications 
> submitted by all users in that queue should be notified of the headroom 
> change.
> * Also today headroom is an absolute number ( I think it should be normalized 
> but then this is going to be not backward compatible..)
> * Also  when admin user refreshes queue headroom has to be updated.
> These all are the potential bugs in headroom calculations

This message was sent by Atlassian JIRA

Reply via email to