[
https://issues.apache.org/jira/browse/YARN-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14325048#comment-14325048
]
Jason Lowe commented on YARN-3136:
----------------------------------
Sorry for the delay in replying. It is true that we could break some
schedulers by moving the definition from a Map to a ConcurrentMap.
Alternatives I see to that approach are:
1) use a read-write lock around accessing the map, which unfortunately can be
wasteful for some map implementations (e.g.: concurrent maps).
2) use methods to access the map whose default implementation use a lock by
default (for backwards compatibility) but can be overridden in derived
schedulers for performance when the underlying map does not require a lock
3) use RTTI when accessing the map to see if it's concurrent and lock when it's
not. Not a huge fan of this one.
> getTransferredContainers can be a bottleneck during AM registration
> -------------------------------------------------------------------
>
> Key: YARN-3136
> URL: https://issues.apache.org/jira/browse/YARN-3136
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
> Affects Versions: 2.6.0
> Reporter: Jason Lowe
> Assignee: Sunil G
> Attachments: 0001-YARN-3136.patch, 0002-YARN-3136.patch
>
>
> While examining RM stack traces on a busy cluster I noticed a pattern of AMs
> stuck waiting for the scheduler lock trying to call getTransferredContainers.
> The scheduler lock is highly contended, especially on a large cluster with
> many nodes heartbeating, and it would be nice if we could find a way to
> eliminate the need to grab this lock during this call. We've already done
> similar work during AM allocate calls to make sure they don't needlessly grab
> the scheduler lock, and it would be good to do so here as well, if possible.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
