[jira] [Commented] (HBASE-14712) MasterProcWALs never clean up

[Elliott Clark (JIRA)]

    [ 
https://issues.apache.org/jira/browse/HBASE-14712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14979407#comment-14979407
 ] 

Elliott Clark commented on HBASE-14712:
---------------------------------------

The CI cluster can't come back up since it ddos' the namenode. However a prod 
cluster that has 200+ master logs

Relevant jstacks:

{code}
Thread 154 (ProcedureExecutor-39):
  State: WAITING
  Blocked count: 0
  Waited count: 5033950324
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 153 (ProcedureExecutor-38):
  State: WAITING
  Blocked count: 1
  Waited count: 5024948293
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 152 (ProcedureExecutor-37):
  State: WAITING
  Blocked count: 10
  Waited count: 5080448452
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 151 (ProcedureExecutor-36):
  State: WAITING
  Blocked count: 4
  Waited count: 5067414781
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 150 (ProcedureExecutor-35):
  State: WAITING
  Blocked count: 406
  Waited count: 5010979283
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 148 (ProcedureExecutor-33):
  State: WAITING
  Blocked count: 0
  Waited count: 5125689465
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 147 (ProcedureExecutor-32):
  State: RUNNABLE
  Blocked count: 5
  Waited count: 5137909716
  Stack:
    sun.misc.Unsafe.unpark(Native Method)
    java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1264)
    java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:151)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 146 (ProcedureExecutor-31):
  State: WAITING
  Blocked count: 11
  Waited count: 4973370758
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 145 (ProcedureExecutor-30):
  State: WAITING
  Blocked count: 9
  Waited count: 4999165364
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 144 (ProcedureExecutor-29):
  State: WAITING
  Blocked count: 0
  Waited count: 4938033019
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 143 (ProcedureExecutor-28):
  State: WAITING
  Blocked count: 6
  Waited count: 5117253857
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 142 (ProcedureExecutor-27):
  State: WAITING
  Blocked count: 0
  Waited count: 4960149681
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 141 (ProcedureExecutor-26):
  State: WAITING
  Blocked count: 4
  Waited count: 5059471446
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 140 (ProcedureExecutor-25):
  State: WAITING
  Blocked count: 0
  Waited count: 4714376204
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 136 (ProcedureExecutor-21):
  State: WAITING
  Blocked count: 7
  Waited count: 5072882873
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 135 (ProcedureExecutor-20):
  State: WAITING
  Blocked count: 0
  Waited count: 5019209711
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 133 (ProcedureExecutor-18):
  State: WAITING
  Blocked count: 0
  Waited count: 4875642244
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 132 (ProcedureExecutor-17):
  State: WAITING
  Blocked count: 5
  Waited count: 5096210372
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 131 (ProcedureExecutor-16):
  State: WAITING
  Blocked count: 0
  Waited count: 5193696305
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 130 (ProcedureExecutor-15):
  State: WAITING
  Blocked count: 1
  Waited count: 5159967893
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 129 (ProcedureExecutor-14):
  State: WAITING
  Blocked count: 1
  Waited count: 4944911560
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 127 (ProcedureExecutor-12):
  State: WAITING
  Blocked count: 3
  Waited count: 4758449709
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 126 (ProcedureExecutor-11):
  State: WAITING
  Blocked count: 0
  Waited count: 4875194485
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 125 (ProcedureExecutor-10):
  State: WAITING
  Blocked count: 8
  Waited count: 5098641806
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 124 (ProcedureExecutor-9):
  State: WAITING
  Blocked count: 5
  Waited count: 4956138868
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 123 (ProcedureExecutor-8):
  State: WAITING
  Blocked count: 8
  Waited count: 4949138759
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 122 (ProcedureExecutor-7):
  State: WAITING
  Blocked count: 3
  Waited count: 5007168959
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 121 (ProcedureExecutor-6):
  State: WAITING
  Blocked count: 2
  Waited count: 5025941067
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 120 (ProcedureExecutor-5):
  State: WAITING
  Blocked count: 2
  Waited count: 5118307254
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 119 (ProcedureExecutor-4):
  State: WAITING
  Blocked count: 2
  Waited count: 4839725179
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 118 (ProcedureExecutor-3):
  State: WAITING
  Blocked count: 1
  Waited count: 5040896840
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 117 (ProcedureExecutor-2):
  State: WAITING
  Blocked count: 14
  Waited count: 5119062593
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 116 (ProcedureExecutor-1):
  State: WAITING
  Blocked count: 12
  Waited count: 5075402180
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:476)
Thread 115 (ProcedureExecutor-0):
  State: WAITING
  Blocked count: 0
  Waited count: 4866205757
  Waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@6e6a5656
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
    
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
    
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
    java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
    
org.apache.hadoop.hbase.master.procedure.MasterProcedureQueue.poll(MasterProcedureQueue.java:133)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:725)
    
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:72)
{code}

> MasterProcWALs never clean up
> -----------------------------
>
>                 Key: HBASE-14712
>                 URL: https://issues.apache.org/jira/browse/HBASE-14712
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Elliott Clark
>            Assignee: Matteo Bertozzi
>            Priority: Blocker
>
> MasterProcWALs directory grows pretty much un-bounded. Because of that when 
> master failover happens the NN is flooded with connections and everything 
> grinds to a halt.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)