[jira] [Updated] (YARN-2874) Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps
[ https://issues.apache.org/jira/browse/YARN-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-2874: -- Fix Version/s: 2.6.1 Pulled this into 2.6.1. Ran compilation before the push. Patch applied cleanly. > Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further > apps > - > > Key: YARN-2874 > URL: https://issues.apache.org/jira/browse/YARN-2874 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0, 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.7.0, 2.6.1 > > Attachments: YARN-2874.20141118-1.patch, YARN-2874.20141118-2.patch > > > When token renewal fails and the application finishes this dead lock can occur > Jstack dump : > {quote} > Found one Java-level deadlock: > = > "DelegationTokenRenewer #181865": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > "DelayedTokenCanceller": > waiting to lock monitor 0x04141718 (object 0xc7eae720, a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask), > which is held by "Timer-4" > "Timer-4": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > > Java stack information for the threads listed above: > === > "DelegationTokenRenewer #181865": > at java.util.Collections$SynchronizedCollection.add(Collections.java:1636) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addTokenToList(DelegationTokenRenewer.java:322) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:398) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "DelayedTokenCanceller": > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.cancel(DelegationTokenRenewer.java:443) > - waiting to lock <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeApplicationFromRenewal(DelegationTokenRenewer.java:558) > - locked <0xc18a9998> (a java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$300(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelayedTokenRemovalRunnable.run(DelegationTokenRenewer.java:599) > at java.lang.Thread.run(Thread.java:745) > "Timer-4": > at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeFailedDelegationToken(DelegationTokenRenewer.java:503) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$100(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:437) > - locked <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > > Found 1 deadlock. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2874) Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps
[ https://issues.apache.org/jira/browse/YARN-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-2874: -- Labels: 2.6.1-candidate (was: ) > Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further > apps > - > > Key: YARN-2874 > URL: https://issues.apache.org/jira/browse/YARN-2874 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0, 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Blocker > Labels: 2.6.1-candidate > Fix For: 2.7.0 > > Attachments: YARN-2874.20141118-1.patch, YARN-2874.20141118-2.patch > > > When token renewal fails and the application finishes this dead lock can occur > Jstack dump : > {quote} > Found one Java-level deadlock: > = > "DelegationTokenRenewer #181865": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > "DelayedTokenCanceller": > waiting to lock monitor 0x04141718 (object 0xc7eae720, a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask), > which is held by "Timer-4" > "Timer-4": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > > Java stack information for the threads listed above: > === > "DelegationTokenRenewer #181865": > at java.util.Collections$SynchronizedCollection.add(Collections.java:1636) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addTokenToList(DelegationTokenRenewer.java:322) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:398) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "DelayedTokenCanceller": > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.cancel(DelegationTokenRenewer.java:443) > - waiting to lock <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeApplicationFromRenewal(DelegationTokenRenewer.java:558) > - locked <0xc18a9998> (a java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$300(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelayedTokenRemovalRunnable.run(DelegationTokenRenewer.java:599) > at java.lang.Thread.run(Thread.java:745) > "Timer-4": > at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeFailedDelegationToken(DelegationTokenRenewer.java:503) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$100(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:437) > - locked <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > > Found 1 deadlock. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2874) Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps
[ https://issues.apache.org/jira/browse/YARN-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated YARN-2874: - Affects Version/s: 2.6.0 > Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further > apps > - > > Key: YARN-2874 > URL: https://issues.apache.org/jira/browse/YARN-2874 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.6.0, 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Blocker > Attachments: YARN-2874.20141118-1.patch, YARN-2874.20141118-2.patch > > > When token renewal fails and the application finishes this dead lock can occur > Jstack dump : > {quote} > Found one Java-level deadlock: > = > "DelegationTokenRenewer #181865": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > "DelayedTokenCanceller": > waiting to lock monitor 0x04141718 (object 0xc7eae720, a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask), > which is held by "Timer-4" > "Timer-4": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > > Java stack information for the threads listed above: > === > "DelegationTokenRenewer #181865": > at java.util.Collections$SynchronizedCollection.add(Collections.java:1636) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addTokenToList(DelegationTokenRenewer.java:322) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:398) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "DelayedTokenCanceller": > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.cancel(DelegationTokenRenewer.java:443) > - waiting to lock <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeApplicationFromRenewal(DelegationTokenRenewer.java:558) > - locked <0xc18a9998> (a java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$300(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelayedTokenRemovalRunnable.run(DelegationTokenRenewer.java:599) > at java.lang.Thread.run(Thread.java:745) > "Timer-4": > at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeFailedDelegationToken(DelegationTokenRenewer.java:503) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$100(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:437) > - locked <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > > Found 1 deadlock. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2874) Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps
[ https://issues.apache.org/jira/browse/YARN-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-2874: --- Target Version/s: 2.7.0 Affects Version/s: (was: 2.4.1) (was: 2.5.0) > Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further > apps > - > > Key: YARN-2874 > URL: https://issues.apache.org/jira/browse/YARN-2874 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Blocker > Attachments: YARN-2874.20141118-1.patch, YARN-2874.20141118-2.patch > > > When token renewal fails and the application finishes this dead lock can occur > Jstack dump : > {quote} > Found one Java-level deadlock: > = > "DelegationTokenRenewer #181865": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > "DelayedTokenCanceller": > waiting to lock monitor 0x04141718 (object 0xc7eae720, a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask), > which is held by "Timer-4" > "Timer-4": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > > Java stack information for the threads listed above: > === > "DelegationTokenRenewer #181865": > at java.util.Collections$SynchronizedCollection.add(Collections.java:1636) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addTokenToList(DelegationTokenRenewer.java:322) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:398) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "DelayedTokenCanceller": > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.cancel(DelegationTokenRenewer.java:443) > - waiting to lock <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeApplicationFromRenewal(DelegationTokenRenewer.java:558) > - locked <0xc18a9998> (a java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$300(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelayedTokenRemovalRunnable.run(DelegationTokenRenewer.java:599) > at java.lang.Thread.run(Thread.java:745) > "Timer-4": > at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeFailedDelegationToken(DelegationTokenRenewer.java:503) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$100(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:437) > - locked <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > > Found 1 deadlock. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2874) Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps
[ https://issues.apache.org/jira/browse/YARN-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-2874: --- Priority: Blocker (was: Critical) > Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further > apps > - > > Key: YARN-2874 > URL: https://issues.apache.org/jira/browse/YARN-2874 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.5.0, 2.4.1, 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Blocker > Attachments: YARN-2874.20141118-1.patch, YARN-2874.20141118-2.patch > > > When token renewal fails and the application finishes this dead lock can occur > Jstack dump : > {quote} > Found one Java-level deadlock: > = > "DelegationTokenRenewer #181865": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > "DelayedTokenCanceller": > waiting to lock monitor 0x04141718 (object 0xc7eae720, a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask), > which is held by "Timer-4" > "Timer-4": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > > Java stack information for the threads listed above: > === > "DelegationTokenRenewer #181865": > at java.util.Collections$SynchronizedCollection.add(Collections.java:1636) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addTokenToList(DelegationTokenRenewer.java:322) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:398) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "DelayedTokenCanceller": > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.cancel(DelegationTokenRenewer.java:443) > - waiting to lock <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeApplicationFromRenewal(DelegationTokenRenewer.java:558) > - locked <0xc18a9998> (a java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$300(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelayedTokenRemovalRunnable.run(DelegationTokenRenewer.java:599) > at java.lang.Thread.run(Thread.java:745) > "Timer-4": > at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeFailedDelegationToken(DelegationTokenRenewer.java:503) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$100(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:437) > - locked <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > > Found 1 deadlock. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2874) Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps
[ https://issues.apache.org/jira/browse/YARN-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-2874: Attachment: YARN-2874.20141118-2.patch Updated patch with fixes for review comment > Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further > apps > - > > Key: YARN-2874 > URL: https://issues.apache.org/jira/browse/YARN-2874 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.5.0, 2.4.1, 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Critical > Attachments: YARN-2874.20141118-1.patch, YARN-2874.20141118-2.patch > > > When token renewal fails and the application finishes this dead lock can occur > Jstack dump : > {quote} > Found one Java-level deadlock: > = > "DelegationTokenRenewer #181865": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > "DelayedTokenCanceller": > waiting to lock monitor 0x04141718 (object 0xc7eae720, a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask), > which is held by "Timer-4" > "Timer-4": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > > Java stack information for the threads listed above: > === > "DelegationTokenRenewer #181865": > at java.util.Collections$SynchronizedCollection.add(Collections.java:1636) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addTokenToList(DelegationTokenRenewer.java:322) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:398) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "DelayedTokenCanceller": > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.cancel(DelegationTokenRenewer.java:443) > - waiting to lock <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeApplicationFromRenewal(DelegationTokenRenewer.java:558) > - locked <0xc18a9998> (a java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$300(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelayedTokenRemovalRunnable.run(DelegationTokenRenewer.java:599) > at java.lang.Thread.run(Thread.java:745) > "Timer-4": > at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeFailedDelegationToken(DelegationTokenRenewer.java:503) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$100(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:437) > - locked <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > > Found 1 deadlock. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2874) Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps
[ https://issues.apache.org/jira/browse/YARN-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naganarasimha G R updated YARN-2874: Attachment: YARN-2874.20141118-1.patch > Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further > apps > - > > Key: YARN-2874 > URL: https://issues.apache.org/jira/browse/YARN-2874 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.5.0, 2.4.1, 2.5.1 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R >Priority: Critical > Attachments: YARN-2874.20141118-1.patch > > > When token renewal fails and the application finishes this dead lock can occur > Jstack dump : > {quote} > Found one Java-level deadlock: > = > "DelegationTokenRenewer #181865": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > "DelayedTokenCanceller": > waiting to lock monitor 0x04141718 (object 0xc7eae720, a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask), > which is held by "Timer-4" > "Timer-4": > waiting to lock monitor 0x00900918 (object 0xc18a9998, a > java.util.Collections$SynchronizedSet), > which is held by "DelayedTokenCanceller" > > Java stack information for the threads listed above: > === > "DelegationTokenRenewer #181865": > at java.util.Collections$SynchronizedCollection.add(Collections.java:1636) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.addTokenToList(DelegationTokenRenewer.java:322) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:398) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$500(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.handleDTRenewerAppSubmitEvent(DelegationTokenRenewer.java:657) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:638) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > "DelayedTokenCanceller": > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.cancel(DelegationTokenRenewer.java:443) > - waiting to lock <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeApplicationFromRenewal(DelegationTokenRenewer.java:558) > - locked <0xc18a9998> (a java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$300(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelayedTokenRemovalRunnable.run(DelegationTokenRenewer.java:599) > at java.lang.Thread.run(Thread.java:745) > "Timer-4": > at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639) > - waiting to lock <0xc18a9998> (a > java.util.Collections$SynchronizedSet) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.removeFailedDelegationToken(DelegationTokenRenewer.java:503) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$100(DelegationTokenRenewer.java:70) > at > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:437) > - locked <0xc7eae720> (a > org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask) > at java.util.TimerThread.mainLoop(Timer.java:555) > at java.util.TimerThread.run(Timer.java:505) > > Found 1 deadlock. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)