[
https://issues.apache.org/jira/browse/YARN-6029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15786636#comment-15786636
]
Naganarasimha G R commented on YARN-6029:
-----------------------------------------
Thanks [~wangda] and [~Tao Yang],
Yes agree with [~wangda] that this is where we need to fix as the cause for the
deadlock is the reverse locking rather than ParentQueue calling the child queue
(as it can be happen in other flows too). And solution looks much simpler than
i initially thought !
bq. And in addition, I suggest to downgrade severity to critical to unblock
2.8, since this only happens rarely.
As solution looks simple and can go in quickly, hope it can get in by 2.8
itself. Based on your previous comment, i presume you also plan the same.
Also checked other places where *getParent()* is called in the LeafQueue and
its call hierarchy, seems like there is no other similar issues causing
deadlock.
> CapacityScheduler deadlock when ParentQueue#getQueueUserAclInfo is called by
> Thread_A at the moment that Thread_B calls LeafQueue#assignContainers to
> release a reserved container
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-6029
> URL: https://issues.apache.org/jira/browse/YARN-6029
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
> Affects Versions: 2.8.0
> Reporter: Tao Yang
> Assignee: Tao Yang
> Priority: Critical
> Attachments: YARN-6029.001.patch, YARN-6029.002.patch, deadlock.jstack
>
>
> When ParentQueue#getQueueUserAclInfo is called (e.g. a client calls
> YarnClient#getQueueAclsInfo) just at the moment that
> LeafQueue#assignContainers is called and before notifying parent queue to
> release resource (should release a reserved container), then ResourceManager
> can deadlock. I found this problem on our testing environment for hadoop2.8.
> Reproduce the deadlock in chronological order
> * 1. Thread A (ResourceManager Event Processor) calls synchronized
> LeafQueue#assignContainers (got LeafQueue instance lock of queue root.a)
> * 2. Thread B (IPC Server handler) calls synchronized
> ParentQueue#getQueueUserAclInfo (got ParentQueue instance lock of queue
> root), iterates over children queue acls and is blocked when calling
> synchronized LeafQueue#getQueueUserAclInfo (the LeafQueue instance lock of
> queue root.a is hold by Thread A)
> * 3. Thread A wants to inform the parent queue that a container is being
> completed and is blocked when invoking synchronized
> ParentQueue#internalReleaseResource method (the ParentQueue instance lock of
> queue root is hold by Thread B)
> I think the synchronized modifier of LeafQueue#getQueueUserAclInfo can be
> removed to solve this problem, since this method appears to not affect fields
> of LeafQueue instance.
> Attach patch with UT for review.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
