Rohith commented on YARN-2946:

Above 2 deadlock's can be directly fixed by removing *synchronized* keyword 
which was not really required. 

But I see there many other potential deadlocks which can appear easily in the 
following methods. Below all methods does reverse locking i.e 
*zkRMStateStore.class -> StateMachine.doTransition()* through 
# Method {{RMStateStore#storeRMDTMasterKey()}}
# Method {{RMStateStore#removeRMDTMasterKey()}}
# Method {{RMStateStore#storeRMDelegationTokenAndSequenceNumber()}}
# Method {{RMStateStore#removeRMDelegationToken()}}
# Method {{RMStateStore#updateRMDelegationTokenAndSequenceNumber()}}
# Method {{ZKRMStateStore#storeOrUpdateAMRMTokenSecretManagerState()}}

So only fixing 1 or 2 deadlock flows does not really fix other potential dead 
lock issues.

*I propose following solution to handle all these deadlock flows*
Option-1 : 
# For all above mentioned method's causing deadlock , introduce StateMachine in 
RMStateStore like handling application store. So all the execution flows from 
# Along with 1st , StateMachine should be guarded with Read-Write lock.

Option-2 : 
# Fix the visible eadlocks i.e 2 found in this jira. And Option-1 do in 
separate improvement task.

Handling all the deadlock flows, i would like to do in one umbrella jira. This 
is to ensure we do not miss any these deadlock flows.

Please let me your suggestions/thoughts?

> Deadlock in ZKRMStateStore
> --------------------------
>                 Key: YARN-2946
>                 URL: https://issues.apache.org/jira/browse/YARN-2946
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.7.0
>            Reporter: Rohith
>            Assignee: Rohith
>            Priority: Blocker
>         Attachments: 0001-YARN-2946.patch, 0002-YARN-2946.patch, 
> TestYARN2946.java
> Found one deadlock in ZKRMStateStore.
> # Initial stage zkClient is null because of zk disconnected event.
> # When ZKRMstatestore#runWithCheck()  wait(zkSessionTimeout) for zkClient to 
> re establish zookeeper connection either via synconnected or expired event, 
> it is highly possible that any other thred can obtain lock on 
> {{ZKRMStateStore.this}} from state machine transition events. This cause 
> Deadlock in ZKRMStateStore.

This message was sent by Atlassian JIRA

Reply via email to