[
https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nkeywal updated HBASE-4724:
---------------------------
Assignee: nkeywal
Status: Patch Available (was: Open)
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
> Key: HBASE-4724
> URL: https://issues.apache.org/jira/browse/HBASE-4724
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 0.94.0
> Reporter: nkeywal
> Assignee: nkeywal
> Priority: Critical
> Attachments: 20111102_4724_TestAdmin.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN [Master:0;localhost,39664,1320187706355]
> master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to
> localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol
> org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28,
> server = 29)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO
> [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80):
> Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on
> Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
>
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
>
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
>
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
>
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
>
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not
> end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal,
> with some threads surprised to have updates that were not written in the wal.
> Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from
> nkeywal):
> State: TIMED_WAITING
> Blocked count: 360
> Waited count: 359
> Stack:
> java.lang.Object.wait(Native Method)
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
> org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
> State: WAITING
> Blocked count: 0
> Waited count: 4
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271
> (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
> State: WAITING
> Blocked count: 217
> Waited count: 174
> Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
>
> org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
> org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
>
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
> org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
> java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
>
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
> java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
> State: WAITING
> Blocked count: 40
> Waited count: 39
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
> State: RUNNABLE
> Blocked count: 55
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1968
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
> State: WAITING
> Blocked count: 7
> Waited count: 1969
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
> State: WAITING
> Blocked count: 11
> Waited count: 1968
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
> State: WAITING
> Blocked count: 6
> Waited count: 1969
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
> State: WAITING
> Blocked count: 3
> Waited count: 1970
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
> java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
> State: WAITING
> Blocked count: 0
> Waited count: 269
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>
> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
> State: WAITING
> Blocked count: 2
> Waited count: 252
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 207
> Stack:
> java.lang.Object.wait(Native Method)
>
> org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
> State: RUNNABLE
> Blocked count: 38
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 407
> Stack:
> java.lang.Thread.sleep(Native Method)
>
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102
> (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
>
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode:
> [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
> State: TIMED_WAITING
> Blocked count: 137
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
>
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0
> SelectChannelConnector@localhost:59858):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101
> (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 98
> (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
>
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
>
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode:
> [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
> State: TIMED_WAITING
> Blocked count: 149
> Waited count: 290
> Stack:
> java.lang.Object.wait(Native Method)
>
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0
> SelectChannelConnector@localhost:36427):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75
> (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 72
> (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 408
> Stack:
> java.lang.Thread.sleep(Native Method)
>
> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
> java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
> State: WAITING
> Blocked count: 0
> Waited count: 1
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
>
> org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
> java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode:
> [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
> State: TIMED_WAITING
> Blocked count: 148
> Waited count: 283
> Stack:
> java.lang.Object.wait(Native Method)
>
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
> java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 15
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0
> SelectChannelConnector@localhost:56535):
> State: RUNNABLE
> Blocked count: 1
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48
> (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 45
> (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 1
> Stack:
> java.lang.Thread.sleep(Native Method)
> org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
> java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
> State: WAITING
> Blocked count: 2
> Waited count: 66
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 64
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
> State: WAITING
> Blocked count: 4
> Waited count: 68
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 65
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
> State: WAITING
> Blocked count: 1
> Waited count: 66
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
> State: WAITING
> Blocked count: 0
> Waited count: 64
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
> State: WAITING
> Blocked count: 3
> Waited count: 65
> Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
> State: RUNNABLE
> Blocked count: 43
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 17
> Stack:
> java.lang.Object.wait(Native Method)
> java.util.TimerThread.mainLoop(Timer.java:509)
> java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0
> SelectChannelConnector@localhost:33079):
> State: RUNNABLE
> Blocked count: 2
> Waited count: 1
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 8
> Stack:
> java.lang.Object.wait(Native Method)
>
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
> State: RUNNABLE
> Blocked count: 45
> Waited count: 44
> Stack:
> sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
> org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> java.lang.Thread.run(Thread.java:662)
> Thread 14
> (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
> State: TIMED_WAITING
> Blocked count: 1
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
>
> org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
> java.lang.Thread.run(Thread.java:662)
> Thread 13
> (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
> State: TIMED_WAITING
> Blocked count: 2
> Waited count: 147
> Stack:
> java.lang.Thread.sleep(Native Method)
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
> java.lang.Thread.run(Thread.java:662)
> Thread 12
> (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 220
> Stack:
> java.lang.Thread.sleep(Native Method)
>
> org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
> java.lang.Thread.run(Thread.java:662)
> Thread 11
> (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 89
> Stack:
> java.lang.Thread.sleep(Native Method)
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
> java.lang.Thread.run(Thread.java:662)
> Thread 10
> (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
> State: TIMED_WAITING
> Blocked count: 0
> Waited count: 2
> Stack:
> java.lang.Thread.sleep(Native Method)
>
> org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
> java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
> State: RUNNABLE
> Blocked count: 0
> Waited count: 0
> Stack:
> Thread 3 (Finalizer):
> State: WAITING
> Blocked count: 9
> Waited count: 10
> Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
> State: WAITING
> Blocked count: 11
> Waited count: 12
> Waiting on java.lang.ref.Reference$Lock@270d75a3
> Stack:
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:485)
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
> State: RUNNABLE
> Blocked count: 104
> Waited count: 97
> Stack:
> sun.management.ThreadImpl.getThreadInfo1(Native Method)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
> sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
>
> org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
>
> org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
> org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
> org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
>
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
>
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
>
> org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> java.lang.reflect.Method.invoke(Method.java:597)
>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
> org.junit.runners.ParentRunner.run(ParentRunner.java:236)
>
> org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
