Wangda Tan commented on YARN-2848:

[~cwelch], Thanks for your explanation, I think it is valid to have such 
mechanism of course :), I just concerned about the cost.

The pull model you mentioned is isomorphic as the push model (send events to 
apps, which we can also add filters to select which apps to send). And wrt pull 
model, we don't have dedicated thread for app to do that. And more problematic, 
if we cannot get apps synchronously handle such events, we need prepare a event 
queue for apps to do that.

And I think the statement is not always true
bq. and would not impose any O ( n ) (where n=#app in cluster) operations on 
any single/synchronous code path
Since it is possible we change labels on a set of nodes (say 1k nodes), and 
many applications could run across the 1k nodes, some operation will scan nodes 
and build information from scratch, it is a O ( n * m ) operation in very 
extreme cases.

> (FICA) Applications should maintain an application specific 'cluster' 
> resource to calculate headroom and userlimit
> ------------------------------------------------------------------------------------------------------------------
>                 Key: YARN-2848
>                 URL: https://issues.apache.org/jira/browse/YARN-2848
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>            Reporter: Craig Welch
>            Assignee: Craig Welch
> Likely solutions to [YARN-1680] (properly handling node and rack blacklisting 
> with cluster level node additions and removals) will entail managing an 
> application-level "slice" of the cluster resource available to the 
> application for use in accurately calculating the application headroom and 
> user limit.  There is an assumption that events which impact this resource 
> will occur less frequently than the need to calculate headroom, userlimit, 
> etc (which is a valid assumption given that occurs per-allocation heartbeat). 
>  Given that, the application should (with assistance from cluster-level 
> code...) detect changes to the composition of the cluster (node addition, 
> removal) and when those have occurred, calculate an application specific 
> cluster resource by comparing cluster nodes to it's own blacklist (both rack 
> and individual node).  I think it makes sense to include nodelabel 
> considerations into this calculation as it will be efficient to do both at 
> the same time and the single resource value reflecting both constraints could 
> then be used for efficient frequent headroom and userlimit calculations while 
> remaining highly accurate.  The application would need to be made aware of 
> nodelabel changes it is interested in (the application or removal of labels 
> of interest to the application to/from nodes).  For this purpose, the 
> application submissions's nodelabel expression would be used to determine the 
> nodelabel impact on the resource used to calculate userlimit and headroom 
> (Cases where the application elected to request resources not using the 
> application level label expression are out of scope for this - but for the 
> common usecase of an application which uses a particular expression 
> throughout, userlimit and headroom would be accurate) This could also provide 
> an overall mechanism for handling application-specific resource constraints 
> which might be added in the future.

This message was sent by Atlassian JIRA

Reply via email to