[
https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494940#comment-14494940
]
Jason Lowe commented on YARN-3487:
----------------------------------
Sample stacktrace of a blocked thread:
{code}
"IPC Server handler 48 on x" daemon prio=10 tid=0x00007fd4991d5000 nid=0x5d53
waiting for monitor entry [0x00007fd45cf1a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getQueueInfo(CapacityScheduler.java:909)
- waiting to lock <0x000000023ae2c938> (a
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:223)
at
org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.validateResourceRequests(RMServerUtils.java:96)
at
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:501)
- locked <0x00000002616389e0> (a
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService$AllocateResponseLock)
at
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1694)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
{code}
In 2.6 we started calling getQueueInfo to validate resource requests so we
could check the labels. Unfortunately getQueueInfo grabs the scheduler lock
unnecessarily:
{code}
public QueueInfo getQueueInfo(String queueName,
boolean includeChildQueues, boolean recursive)
throws IOException {
CSQueue queue = null;
synchronized (this) {
queue = this.queues.get(queueName);
}
if (queue == null) {
throw new IOException("Unknown queue: " + queueName);
}
return queue.getQueueInfo(includeChildQueues, recursive);
}
{code}
this.queues is a ConcurrentHashMap, so there's not much utility in grabbing the
lock just to lookup something in that map. Worse, it adds a lot of contention
on an already highly contentious lock.
> CapacityScheduler.getQueueInfo grabs scheduler lock unnecessarily
> -----------------------------------------------------------------
>
> Key: YARN-3487
> URL: https://issues.apache.org/jira/browse/YARN-3487
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
> Affects Versions: 2.6.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Critical
>
> Recently saw a significant slowdown of applications on a large cluster, and
> we noticed there were a large number of blocked threads on the RM. Most of
> the blocked threads were waiting for the CapacityScheduler lock while calling
> getQueueInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
