TestAdmin hangs randomly in trunk
---------------------------------
Key: HBASE-4724
URL: https://issues.apache.org/jira/browse/HBASE-4724
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.94.0
Reporter: nkeywal
Priority: Critical
fom the logs in my env
{noformat}
2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355]
master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to
localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol
org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28,
server = 29)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
{noformat}
Anyway, after this the logs finishes with:
{noformat}
2011-11-01 15:54:35,132 INFO
[Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80):
Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on
Master:0;localhost,39664,1320187706355
{noformat}
it's in
{noformat}
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
{noformat}
So that's at least why adding a timeout wont help and may be why it does not
end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
I also wonder if the root cause of the non ending is my modif on the wal, with
some threads surprised to have updates that were not written in the wal. Here
is the full stack dump:
{noformat}
Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from
nkeywal):
State: TIMED_WAITING
Blocked count: 360
Waited count: 359
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
State: WAITING
Blocked count: 0
Waited count: 4
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
State: RUNNABLE
Blocked count: 2
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 152 (Master:0;localhost,39664,1320187706355):
State: WAITING
Blocked count: 217
Waited count: 174
Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
java.lang.Thread.run(Thread.java:662)
Thread 165 (LruBlockCache.EvictionThread):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
java.lang.Thread.run(Thread.java:662)
Thread 151 (main-EventThread):
State: WAITING
Blocked count: 40
Waited count: 39
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 150 (main-SendThread(localhost:21819)):
State: RUNNABLE
Blocked count: 55
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 149 (IPC Server handler 4 on 39664):
State: WAITING
Blocked count: 3
Waited count: 1968
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 148 (IPC Server handler 3 on 39664):
State: WAITING
Blocked count: 7
Waited count: 1969
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 147 (IPC Server handler 2 on 39664):
State: WAITING
Blocked count: 11
Waited count: 1968
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 146 (IPC Server handler 1 on 39664):
State: WAITING
Blocked count: 6
Waited count: 1969
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 145 (IPC Server handler 0 on 39664):
State: WAITING
Blocked count: 3
Waited count: 1970
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 133 (IPC Server listener on 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
Thread 144 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
Thread 143 (IPC Reader 9 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 142 (IPC Reader 8 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 141 (IPC Reader 7 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 140 (IPC Reader 6 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 139 (IPC Reader 5 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 138 (IPC Reader 4 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 137 (IPC Reader 3 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 136 (IPC Reader 2 on port 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 135 (IPC Reader 1 on port 39664):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 134 (IPC Reader 0 on port 39664):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 124 (LeaseChecker):
State: TIMED_WAITING
Blocked count: 0
Waited count: 407
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
java.lang.Thread.run(Thread.java:662)
Thread 121 (ProcessThread:-1):
State: WAITING
Blocked count: 0
Waited count: 269
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
Thread 120 (SyncThread:0):
State: WAITING
Blocked count: 2
Waited count: 252
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
Thread 119 (SessionTracker):
State: TIMED_WAITING
Blocked count: 0
Waited count: 207
Stack:
java.lang.Object.wait(Native Method)
org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
State: RUNNABLE
Blocked count: 38
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
State: TIMED_WAITING
Blocked count: 0
Waited count: 407
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 113 (IPC Server handler 2 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 112 (IPC Server handler 1 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 111 (IPC Server handler 0 on 39899):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 106 (IPC Server listener on 39899):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 108 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 109 (DataNode:
[/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
State: TIMED_WAITING
Blocked count: 137
Waited count: 283
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 107 (pool-4-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 105 (Timer-3):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0
SelectChannelConnector@localhost:59858):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 103 (1185716172@qtp-1197660496-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 101
(refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 98
(refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
State: TIMED_WAITING
Blocked count: 0
Waited count: 408
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 86 (IPC Server handler 2 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 85 (IPC Server handler 1 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 84 (IPC Server handler 0 on 56098):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 80 (IPC Server listener on 56098):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 82 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 83 (DataNode:
[/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
State: TIMED_WAITING
Blocked count: 149
Waited count: 290
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 81 (pool-3-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 79 (Timer-2):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 78 (348878159@qtp-164967086-1 - Acceptor0
SelectChannelConnector@localhost:36427):
State: RUNNABLE
Blocked count: 2
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 77 (1905729203@qtp-164967086-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 75
(refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 72
(refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
State: TIMED_WAITING
Blocked count: 0
Waited count: 408
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
java.lang.Thread.run(Thread.java:662)
Thread 60 (IPC Server handler 2 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 59 (IPC Server handler 1 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 58 (IPC Server handler 0 on 40388):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 53 (IPC Server listener on 40388):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 55 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
java.lang.Thread.run(Thread.java:662)
Thread 57 (DataNode:
[/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
State: TIMED_WAITING
Blocked count: 148
Waited count: 283
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
java.lang.Thread.run(Thread.java:662)
Thread 54 (pool-2-thread-1):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 52 (Timer-1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 15
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0
SelectChannelConnector@localhost:56535):
State: RUNNABLE
Blocked count: 1
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 50 (1191055321@qtp-1068922666-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 48
(refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 45
(refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
java.lang.Thread.run(Thread.java:662)
Thread 31 (IPC Server handler 9 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 30 (IPC Server handler 8 on 52227):
State: WAITING
Blocked count: 3
Waited count: 65
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 29 (IPC Server handler 7 on 52227):
State: WAITING
Blocked count: 2
Waited count: 66
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 28 (IPC Server handler 6 on 52227):
State: WAITING
Blocked count: 1
Waited count: 64
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 27 (IPC Server handler 5 on 52227):
State: WAITING
Blocked count: 4
Waited count: 68
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 26 (IPC Server handler 4 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 25 (IPC Server handler 3 on 52227):
State: WAITING
Blocked count: 0
Waited count: 65
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 24 (IPC Server handler 2 on 52227):
State: WAITING
Blocked count: 1
Waited count: 66
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 23 (IPC Server handler 1 on 52227):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 22 (IPC Server handler 0 on 52227):
State: WAITING
Blocked count: 3
Waited count: 65
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 15 (IPC Server listener on 52227):
State: RUNNABLE
Blocked count: 43
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 17 (IPC Server Responder):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 21 (Timer-0):
State: TIMED_WAITING
Blocked count: 1
Waited count: 17
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:509)
java.util.TimerThread.run(Timer.java:462)
Thread 20 (245022965@qtp-589734556-1 - Acceptor0
SelectChannelConnector@localhost:33079):
State: RUNNABLE
Blocked count: 2
Waited count: 1
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 19 (1821050251@qtp-589734556-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
java.lang.Object.wait(Native Method)
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 16 (pool-1-thread-1):
State: RUNNABLE
Blocked count: 45
Waited count: 44
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
Thread 14
(org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
State: TIMED_WAITING
Blocked count: 1
Waited count: 147
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
java.lang.Thread.run(Thread.java:662)
Thread 13
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
State: TIMED_WAITING
Blocked count: 2
Waited count: 147
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
java.lang.Thread.run(Thread.java:662)
Thread 12
(org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
State: TIMED_WAITING
Blocked count: 0
Waited count: 220
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
java.lang.Thread.run(Thread.java:662)
Thread 11
(org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
State: TIMED_WAITING
Blocked count: 0
Waited count: 89
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
java.lang.Thread.run(Thread.java:662)
Thread 10
(org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
State: TIMED_WAITING
Blocked count: 0
Waited count: 2
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
java.lang.Thread.run(Thread.java:662)
Thread 4 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 9
Waited count: 10
Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 11
Waited count: 12
Waiting on java.lang.ref.Reference$Lock@270d75a3
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:485)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
State: RUNNABLE
Blocked count: 104
Waited count: 97
Stack:
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
org.junit.runners.ParentRunner.run(ParentRunner.java:236)
org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
