[ 
https://issues.apache.org/jira/browse/YARN-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14232317#comment-14232317
 ] 

Karthik Kambatla commented on YARN-2874:
----------------------------------------

For purposes of fixing the deadlock, the patch looks good to me. I am not sure 
how easy it is to add a test for this, so I am okay with not including one.

That said, there is still a race along the lines of the one discusses in 
MAPREDUCE-5384. We should at least create a follow-up JIRA to fix this; Sid's 
suggestion - 
https://issues.apache.org/jira/browse/MAPREDUCE-5384?focusedCommentId=13745421&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13745421
 - should work.

> Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further 
> apps
> ---------------------------------------------------------------------------------
>
>                 Key: YARN-2874
>                 URL: https://issues.apache.org/jira/browse/YARN-2874
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: resourcemanager
>    Affects Versions: 2.5.1
>            Reporter: Naganarasimha G R
>            Assignee: Naganarasimha G R
>            Priority: Blocker
>         Attachments: YARN-2874.20141118-1.patch, YARN-2874.20141118-2.patch
>
>
> When token renewal fails and the application finishes this dead lock can occur
> Jstack dump :
> {quote}
> Found one Java-level deadlock:
> =============================
> "DelegationTokenRenewer #181865":
>   waiting to lock monitor 0x0000000000900918 (object 0x00000000c18a9998, a 
> java.util.Collections$SynchronizedSet),
>   which is held by "DelayedTokenCanceller"
> "DelayedTokenCanceller":
>   waiting to lock monitor 0x0000000004141718 (object 0x00000000c7eae720, a 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask),
>   which is held by "Timer-4"
> "Timer-4":
>   waiting to lock monitor 0x0000000000900918 (object 0x00000000c18a9998, a 
> java.util.Collections$SynchronizedSet),
>   which is held by "DelayedTokenCanceller"
>  
> Java stack information for the threads listed above:
> ===================================================
> "DelegationTokenRenewer #181865":
> at java.util.Collections$SynchronizedCollection.add(Collections.java:1636)
> - waiting to lock <0x00000000c18a9998> (a 
> java.util.Collections$SynchronizedSet)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addTokenToList(DelegationTokenRenewer.java:322)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:398)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> "DelayedTokenCanceller":
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.cancel(DelegationTokenRenewer.java:443)
> - waiting to lock <0x00000000c7eae720> (a 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeApplicationFromRenewal(DelegationTokenRenewer.java:558)
> - locked <0x00000000c18a9998> (a java.util.Collections$SynchronizedSet)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$300(DelegationTokenRenewer.java:70)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelayedTokenRemovalRunnable.run(DelegationTokenRenewer.java:599)
> at java.lang.Thread.run(Thread.java:745)
> "Timer-4":
> at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639)
> - waiting to lock <0x00000000c18a9998> (a 
> java.util.Collections$SynchronizedSet)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeFailedDelegationToken(DelegationTokenRenewer.java:503)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$100(DelegationTokenRenewer.java:70)
> at 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:437)
> - locked <0x00000000c7eae720> (a 
> org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
>  
> Found 1 deadlock.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to