Craig Welch commented on YARN-1680:

bq. This requires when a node doing heartbeat with changed available resource, 
all apps blacklisted the node need to be notified

Well, that's not quite so.  From what we were talking about, it means that the 
blacklist deduction can't be a fixed amount but that it needs to be calculated 
by looking at the unused resource of the blacklisted nodes during headroom 
calculation.  The rest of the above proposal for detecting changes, etc, works, 
but instead of a static deduction value we would need a reference to the 
blacklisted nodes for the app and look at their unused resources during the 
apps headroom calculation, so there is that cost, but it's not related to the 
heartbeat or a notification as such

bq. headroom for app could be under estimated

I think, generally, we should not take an approach which will 
underestimate/underutilize if we have 6302 to fall back on.  If we don't, then 
we might want to do it only if we decide not to do the accurate calculation in 
some cases based on limits (see immediately below), but not as a matter of 

bq. Only do accurate headroom calculation when there're not too much 
blacklisted nodes as well as apps with blacklisted nodes.

I think if we put a limit on it, it should be a purely local decision, to only 
do the calculation with < x blacklisted nodes for an app, which we would expect 
to rarely be an issue.  There is a potential for performance issues here, but 
we don't really know how great a concern it is.

bq.  MAPREDUCE-6302 is targeting to preempt reducer even if we reported 
inaccurate headroom for apps. I think the approach looks good to me

I think that may work as a fallback option for MR, assuming it works out 
without issue, if we decide to not do the proper headroom calculation in some 
cases, but that's MR specific so it won't help non MR apps, and it has the 
issues I brought up before with performance degradation vs the proper headroom 
calculation.  For these reasons I don't think it's a substitute for fixing this 
issue overall, it may be a fallback option if we limit the cases where we do 
the proper adjustment.

bq. Move headroom calculation to application side, I think now we cannot do it 
at least for now...Application will only receive updated NodeReport from when 
node changes heathy status instead of regular heartbeat

Well, in some sense that works OK for this because we really only need to know 
about those changes in node status status wrt the blacklist to detect 
recalculation changes with the approach proposed above.  The problem is that we 
will also need a way to query for current usage per node while doing the 
calculation, I don't know if an efficient call for that exists (it would 
ideally be batch for N nodes where we would ask for all the blacklisted nodes 
at once.)  There is also the broader issue that we don't seem to have a single 
entry point client-side for doing this right now, so we would need to touch a 
few points to add a library/something of that nature to do this, and for AM's 
we may not be aware of/that are not part of the core, they would have to 
potentially do some integration to get this.

> availableResources sent to applicationMaster in heartbeat should exclude 
> blacklistedNodes free memory.
> ------------------------------------------------------------------------------------------------------
>                 Key: YARN-1680
>                 URL: https://issues.apache.org/jira/browse/YARN-1680
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>    Affects Versions: 2.2.0, 2.3.0
>         Environment: SuSE 11 SP2 + Hadoop-2.3 
>            Reporter: Rohith
>            Assignee: Craig Welch
>         Attachments: YARN-1680-WIP.patch, YARN-1680-v2.patch, 
> YARN-1680-v2.patch, YARN-1680.patch
> There are 4 NodeManagers with 8GB each.Total cluster capacity is 32GB.Cluster 
> slow start is set to 1.
> Job is running reducer task occupied 29GB of cluster.One NodeManager(NM-4) is 
> become unstable(3 Map got killed), MRAppMaster blacklisted unstable 
> NodeManager(NM-4). All reducer task are running in cluster now.
> MRAppMaster does not preempt the reducers because for Reducer preemption 
> calculation, headRoom is considering blacklisted nodes memory. This makes 
> jobs to hang forever(ResourceManager does not assing any new containers on 
> blacklisted nodes but returns availableResouce considers cluster free 
> memory). 

This message was sent by Atlassian JIRA

Reply via email to