Jian He commented on YARN-3136:

bq. Speaking of accessing the applications map without holding a proper lock, 
doesn't AbstractYarnScheduler.createReleaseCache() do exactly that?
createReleaseCache is only called In serviceInit, so I think should be fine.

I have a general question that,  is AbstractYarnScheduler supposed to be public 
for external use ?  I think AbstractYarnScheduler is just a common base class 
so as to avoid code duplication among CS/Fair/Fifo schedulers. we'll most 
likely add more changes in this class.  IMHO,  to not complicate things, we can 
just mark AbstractYarnScheduler as Private/Unstable.  this method was added a 
year earlier and now will be invoked only if work-preserving-am-restart is 
enabled. Maybe I'm wrong, I doubt we'll see compatibility issues in reality. 

> getTransferredContainers can be a bottleneck during AM registration
> -------------------------------------------------------------------
>                 Key: YARN-3136
>                 URL: https://issues.apache.org/jira/browse/YARN-3136
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: scheduler
>    Affects Versions: 2.6.0
>            Reporter: Jason Lowe
>            Assignee: Sunil G
>         Attachments: 0001-YARN-3136.patch, 0002-YARN-3136.patch, 
> 0003-YARN-3136.patch, 0004-YARN-3136.patch
> While examining RM stack traces on a busy cluster I noticed a pattern of AMs 
> stuck waiting for the scheduler lock trying to call getTransferredContainers. 
>  The scheduler lock is highly contended, especially on a large cluster with 
> many nodes heartbeating, and it would be nice if we could find a way to 
> eliminate the need to grab this lock during this call.  We've already done 
> similar work during AM allocate calls to make sure they don't needlessly grab 
> the scheduler lock, and it would be good to do so here as well, if possible.

This message was sent by Atlassian JIRA

Reply via email to