Nice job Tatsuya-san. Even on small clusters I raise hbase.regionserver.handler.count to 100 and recommend this in general. I added a comment about this on the troubleshooting page of the HBase wiki: http://wiki.apache.org/hadoop/Hbase/Troubleshooting
Also I filed an issue on this to track it: https://issues.apache.org/jira/browse/HBASE-1910 Best regards, - Andy ________________________________ From: Tatsuya Kawano <[email protected]> To: [email protected] Sent: Fri, October 16, 2009 8:50:38 AM Subject: Re: TableIndexed extreme performance degredation? Hi Kevin, It seems you are having a deadlock issue on your indexed region server. I tried to reproduce this -- 50 client threads who try to put millions of records, autoFlush(false), flushCommits() on every 5,000 put. After inserting about 3 million records, a deadlock occurred on a region server who has both the table and index regions loaded. I have attached a full thread dump of the deadlocked region server, and you can see IPC Server handlers are blocked in org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex(). I found the flowing FIXME comment on updateIndex() method, and it seems this is the deadlock I'm having. ===================================== // FIXME: This call takes place in an RPC, and requires an RPC. This makes for // a likely deadlock if the number of RPCs we are trying to serve is >= the // number of handler threads. private void updateIndex(IndexSpecification indexSpec, byte[] row, SortedMap<byte[], byte[]> columnValues) throws IOException { ===================================== I use HBase 0.20.1 and my region servers were running with 10 RPC handler threads on each (default). Maybe you can workaround this by adding more RPC handlers (increase the value of "hbase.regionserver.handler.count" in hbase-site.xml) I'm going to run the same test again with 70 RPC handlers, and let you know the result. Thanks, Tatsuya -- Tatsuya Kawano (Mr.) Tokyo, Japan On Thu, Oct 15, 2009 at 2:30 AM, Kevin Peterson <[email protected]> wrote: > I'm trying to load data into a table from a Hadoop map job. I have a main > table that stores an average of about 2k per row, and I want to have two > additional index tables, which index 10-20byte keys in the primary table. I > have used TableIndexed and it worked beautifully on small scale testing. > > When I tried to use it at a larger scale, it seems to just freeze up. I see > the Hadoop jobs get through maybe 2.5 million records at a good pace, and > then they just hang. Eventually Hadoop kills the jobs after they haven't > responded for 40 minutes. I don't see anything in the logs (though I > wouldn't know what to look for). > > In comparison, when I remove the TableIndexed region server from > hbase-site.xml, I'm able to easily load my full batch of 12 million records > in an hour. > > Details of cluster: > 1 node ZooKeeper and HBase Master > 4 nodes ZooKeeper, Region Server and DataNode > 4 hadoop datanode / tasktrackers with 3 map slots each > 1 hadoop namenode and jobtracker > > All nodes are EC2 large instances, 2 cores, 8GB ram, two local 500GB disks. > > I have not tuned any memory or performance related settings. I turn on > TableIndexed by setting hbase.regionserver.class to > org.apache.hadoop.hbase.ipc.IndexedRegionInterface and > hbase.regionserver.impl to > org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegionServer. I'm > using HBase 20.1 RC1, with transactional jar compiled from 0.20.0 with > HBASE-1885, which includes my index key creator. > > The behavior makes me think it's something like I need to call commit, but I > can't find anything mentioned. Any ideas? ===================================== 2009-10-16 09:12:53 Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.1-b02-90 mixed mode): "JMX server connection timeout 377" daemon prio=9 tid=0x00000001550b3000 nid=0x158ec6000 in Object.wait() [0x0000000158ec5000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) - locked <0x0000000107fa1728> (a [I) at java.lang.Thread.run(Thread.java:637) Locked ownable synchronizers: - None "RMI Scheduler(0)" daemon prio=9 tid=0x000000014f85a800 nid=0x158dc3000 waiting on condition [0x0000000158dc2000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000107fa00d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:637) Locked ownable synchronizers: - None "RMI TCP Connection(1)-172.16.80.26" daemon prio=9 tid=0x0000000106b31000 nid=0x158bb6000 runnable [0x0000000158bb5000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) - locked <0x0000000107fa3898> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(FilterInputStream.java:66) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:637) Locked ownable synchronizers: - <0x0000000108035bd0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "RMI TCP Accept-0" daemon prio=9 tid=0x00000001530ca000 nid=0x156901000 runnable [0x0000000156900000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) - locked <0x0000000107fa0330> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) at java.lang.Thread.run(Thread.java:637) Locked ownable synchronizers: - None "Attach Listener" daemon prio=9 tid=0x00000001548ca000 nid=0x157a83000 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "IPC Client (47) connection to /172.16.80.26:60022 from an unknown user" daemon prio=5 tid=0x000000015620a000 nid=0x158fc9000 runnable [0x0000000158fc8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x0000000119b56d60> (a sun.nio.ch.Util$1) - locked <0x0000000119b56d48> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000124398148> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) at java.io.FilterInputStream.read(FilterInputStream.java:116) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:279) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) - locked <0x000000011899bf40> (a java.io.BufferedInputStream) at java.io.DataInputStream.readInt(DataInputStream.java:370) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:504) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:448) Locked ownable synchronizers: - None "ResponseProcessor for block blk_6354822949724586880_4142" daemon prio=5 tid=0x000000010295b000 nid=0x156c5b000 runnable [0x0000000156c5a000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000012411b920> (a sun.nio.ch.Util$1) - locked <0x000000012411b908> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000126e55740> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) at java.io.DataInputStream.readFully(DataInputStream.java:178) at java.io.DataInputStream.readLong(DataInputStream.java:399) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2399) Locked ownable synchronizers: - None "DataStreamer for file /hbase/.logs/172.16.80.26,60022,1255650242212/hlog.dat.1255651572580 block blk_6354822949724586880_4142" daemon prio=5 tid=0x0000000106b30000 nid=0x156a04000 in Object.wait() [0x0000000156a03000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2247) - locked <0x0000000126088e70> (a java.util.LinkedList) Locked ownable synchronizers: - None "IPC Server handler 8 on 60022-EventThread" daemon prio=5 tid=0x000000014f845000 nid=0x158cc0000 waiting on condition [0x0000000158cbf000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000001105843c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:414) Locked ownable synchronizers: - None "IPC Server handler 8 on 60022-SendThread" daemon prio=5 tid=0x000000014f844800 nid=0x156b16000 runnable [0x0000000156b15000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000001105957e0> (a sun.nio.ch.Util$1) - locked <0x00000001105957f8> (a java.util.Collections$UnmodifiableSet) - locked <0x000000011015cef0> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:921) Locked ownable synchronizers: - None "pool-1-thread-1" prio=5 tid=0x000000014f95a000 nid=0x158ab3000 waiting on condition [0x0000000158ab2000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000110552968> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:637) Locked ownable synchronizers: - None "LruBlockCache.EvictionThread" daemon prio=5 tid=0x000000014f959800 nid=0x1589b0000 in Object.wait() [0x00000001589af000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:512) - locked <0x00000001103316c0> (a org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread) Locked ownable synchronizers: - None "Transactional leases" daemon prio=5 tid=0x00000001530f1800 nid=0x1588ad000 waiting on condition [0x00000001588ac000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000110597a40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201) at org.apache.hadoop.hbase.Leases.run(Leases.java:81) Locked ownable synchronizers: - None "regionserver/172.16.80.26:60022.oldTransactionCleaner" daemon prio=5 tid=0x00000001530f0800 nid=0x1587aa000 waiting on condition [0x00000001587a9000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) at org.apache.hadoop.hbase.Chore.run(Chore.java:76) Locked ownable synchronizers: - None "IPC Server handler 9 on 60022" daemon prio=5 tid=0x00000001530f0000 nid=0x1586a7000 waiting for monitor entry [0x00000001586a6000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:466) - waiting to lock <0x00000001105852a0> (a org.apache.hadoop.hbase.client.HTable) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex(IndexedRegion.java:213) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndexes(IndexedRegion.java:145) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.put(IndexedRegion.java:96) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208) at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Locked ownable synchronizers: - None "IPC Server handler 8 on 60022" daemon prio=5 tid=0x00000001530ef000 nid=0x1585a4000 waiting for monitor entry [0x00000001585a3000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:466) - waiting to lock <0x00000001105852a0> (a org.apache.hadoop.hbase.client.HTable) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex(IndexedRegion.java:213) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndexes(IndexedRegion.java:145) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.put(IndexedRegion.java:96) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208) at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Locked ownable synchronizers: - None "IPC Server handler 7 on 60022" daemon prio=5 tid=0x00000001530ee800 nid=0x1584a1000 waiting for monitor entry [0x00000001584a0000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:466) - waiting to lock <0x00000001105852a0> (a org.apache.hadoop.hbase.client.HTable) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex(IndexedRegion.java:213) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndexes(IndexedRegion.java:145) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.put(IndexedRegion.java:96) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208) at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Locked ownable synchronizers: - None "IPC Server handler 6 on 60022" daemon prio=5 tid=0x00000001530ed800 nid=0x15839e000 waiting for monitor entry [0x000000015839d000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:466) - waiting to lock <0x00000001105852a0> (a org.apache.hadoop.hbase.client.HTable) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex(IndexedRegion.java:213) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndexes(IndexedRegion.java:145) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.put(IndexedRegion.java:96) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208) at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Locked ownable synchronizers: - None "IPC Server handler 5 on 60022" daemon prio=5 tid=0x00000001530ed000 nid=0x15829b000 in Object.wait() [0x0000000158299000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:717) - locked <0x0000000121595ad0> (a org.apache.hadoop.hbase.ipc.HBaseClient$Call) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:328) at $Proxy6.getProtocolVersion(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:484) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:460) at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:507) at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:418) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:864) - locked <0x00000001105cc1e8> (a java.util.concurrent.ConcurrentHashMap) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getHRegionConnection(HConnectionManager.java:880) at org.apache.hadoop.hbase.client.ServerCallable.instantiateServer(ServerCallable.java:58) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:987) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.doCall(HConnectionManager.java:1192) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1114) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1200) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:605) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:470) - locked <0x00000001105852a0> (a org.apache.hadoop.hbase.client.HTable) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex(IndexedRegion.java:213) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndexes(IndexedRegion.java:145) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.put(IndexedRegion.java:96) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208) at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Locked ownable synchronizers: - None "IPC Server handler 4 on 60022" daemon prio=5 tid=0x00000001530ec000 nid=0x158198000 waiting for monitor entry [0x0000000158197000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:466) - waiting to lock <0x00000001105852a0> (a org.apache.hadoop.hbase.client.HTable) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex(IndexedRegion.java:213) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndexes(IndexedRegion.java:145) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.put(IndexedRegion.java:96) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208) at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Locked ownable synchronizers: - None "IPC Server handler 3 on 60022" daemon prio=5 tid=0x00000001530eb800 nid=0x158095000 waiting for monitor entry [0x0000000158094000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:466) - waiting to lock <0x00000001105852a0> (a org.apache.hadoop.hbase.client.HTable) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex(IndexedRegion.java:213) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndexes(IndexedRegion.java:145) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.put(IndexedRegion.java:96) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208) at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Locked ownable synchronizers: - None "IPC Server handler 2 on 60022" daemon prio=5 tid=0x00000001530ea800 nid=0x157f92000 waiting for monitor entry [0x0000000157f91000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:466) - waiting to lock <0x00000001105852a0> (a org.apache.hadoop.hbase.client.HTable) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex(IndexedRegion.java:213) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndexes(IndexedRegion.java:145) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.put(IndexedRegion.java:96) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208) at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Locked ownable synchronizers: - None "IPC Server handler 1 on 60022" daemon prio=5 tid=0x00000001530ea000 nid=0x157e8f000 waiting for monitor entry [0x0000000157e8e000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:466) - waiting to lock <0x00000001105852a0> (a org.apache.hadoop.hbase.client.HTable) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex(IndexedRegion.java:213) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndexes(IndexedRegion.java:145) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.put(IndexedRegion.java:96) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208) at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Locked ownable synchronizers: - None "IPC Server handler 0 on 60022" daemon prio=5 tid=0x00000001530e9000 nid=0x157d8c000 waiting for monitor entry [0x0000000157d8b000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:466) - waiting to lock <0x00000001105852a0> (a org.apache.hadoop.hbase.client.HTable) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndex(IndexedRegion.java:213) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.updateIndexes(IndexedRegion.java:145) at org.apache.hadoop.hbase.regionserver.tableindexed.IndexedRegion.put(IndexedRegion.java:96) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1208) at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1831) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Locked ownable synchronizers: - None "IPC Server listener on 60022" daemon prio=5 tid=0x00000001530e8800 nid=0x157c89000 runnable [0x0000000157c88000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x0000000110365750> (a sun.nio.ch.Util$1) - locked <0x0000000110365738> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000110224398> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308) Locked ownable synchronizers: - None "IPC Server Responder" daemon prio=5 tid=0x0000000153121000 nid=0x157b86000 runnable [0x0000000157b85000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x0000000110365678> (a sun.nio.ch.Util$1) - locked <0x0000000110365660> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000110224320> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:477) Locked ownable synchronizers: - None "Timer-0" daemon prio=5 tid=0x00000001530d9800 nid=0x157980000 in Object.wait() [0x000000015797f000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x000000011058ecf0> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "1034011...@qtp0-1 - Acceptor0 [email protected]:60032" prio=5 tid=0x0000000154988000 nid=0x15787d000 runnable [0x000000015787c000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00000001105a80d8> (a sun.nio.ch.Util$1) - locked <0x00000001105a80f0> (a java.util.Collections$UnmodifiableSet) - locked <0x000000011016ded0> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) Locked ownable synchronizers: - None "449580...@qtp0-0" prio=5 tid=0x000000015486e000 nid=0x15777a000 in Object.wait() [0x0000000157779000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:565) - locked <0x00000001104823f0> (a org.mortbay.thread.QueuedThreadPool$PoolThread) Locked ownable synchronizers: - None "regionserver/172.16.80.26:60022.leaseChecker" prio=5 tid=0x000000015309b000 nid=0x1574ea000 waiting on condition [0x00000001574e9000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000110139fe0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201) at org.apache.hadoop.hbase.Leases.run(Leases.java:81) Locked ownable synchronizers: - None "regionserver/172.16.80.26:60022.majorCompactionChecker" daemon prio=5 tid=0x000000015313e000 nid=0x1573e7000 waiting on condition [0x00000001573e6000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) at org.apache.hadoop.hbase.Chore.run(Chore.java:76) Locked ownable synchronizers: - None "regionserver/172.16.80.26:60022.worker" daemon prio=5 tid=0x000000015313d000 nid=0x1572e4000 waiting on condition [0x00000001572e3000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000001101386e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1481) at java.lang.Thread.run(Thread.java:637) Locked ownable synchronizers: - None "regionserver/172.16.80.26:60022.compactor" daemon prio=5 tid=0x000000015313c800 nid=0x1571e1000 waiting on condition [0x00000001571e0000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000110379960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:96) Locked ownable synchronizers: - None "regionserver/172.16.80.26:60022.cacheFlusher" daemon prio=5 tid=0x000000015313b800 nid=0x1570de000 waiting on condition [0x00000001570dd000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000110379870> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:145) Locked ownable synchronizers: - None "regionserver/172.16.80.26:60022.logFlusher" daemon prio=5 tid=0x000000015313b000 nid=0x156fdb000 waiting on condition [0x0000000156fda000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) at org.apache.hadoop.hbase.Chore.run(Chore.java:76) Locked ownable synchronizers: - None "regionserver/172.16.80.26:60022.logRoller" daemon prio=5 tid=0x000000015313a000 nid=0x156ed8000 in Object.wait() [0x0000000156ed7000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:65) - locked <0x0000000110139b30> (a java.util.concurrent.atomic.AtomicBoolean) Locked ownable synchronizers: - None "LeaseChecker" daemon prio=5 tid=0x000000014f846000 nid=0x156dd5000 waiting on condition [0x0000000156dd4000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066) at java.lang.Thread.run(Thread.java:637) Locked ownable synchronizers: - None "IPC Client (47) connection to /172.16.80.26:60000 from an unknown user" daemon prio=5 tid=0x000000014f82d000 nid=0x153f5f000 in Object.wait() [0x0000000153f5e000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:404) - locked <0x00000001101c0ee0> (a org.apache.hadoop.hbase.ipc.HBaseClient$Connection) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:447) Locked ownable synchronizers: - None "DestroyJavaVM" prio=5 tid=0x0000000154866800 nid=0x100501000 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "regionserver/172.16.80.26:60022" prio=5 tid=0x0000000154866000 nid=0x153e07000 waiting on condition [0x0000000153e06000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:591) at java.lang.Thread.run(Thread.java:637) Locked ownable synchronizers: - None "main-EventThread" daemon prio=5 tid=0x000000015502c800 nid=0x153d04000 waiting on condition [0x0000000153d03000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000110139388> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:414) Locked ownable synchronizers: - None "main-SendThread" daemon prio=5 tid=0x000000015502b800 nid=0x153c01000 runnable [0x0000000153c00000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x000000011016d2a0> (a sun.nio.ch.Util$1) - locked <0x000000011016d2b8> (a java.util.Collections$UnmodifiableSet) - locked <0x000000011015c2c0> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:921) Locked ownable synchronizers: - None "Low Memory Detector" daemon prio=5 tid=0x000000014f859000 nid=0x151b04000 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "CompilerThread1" daemon prio=9 tid=0x000000014f858000 nid=0x151a01000 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "CompilerThread0" daemon prio=9 tid=0x000000014f857800 nid=0x150f07000 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Signal Dispatcher" daemon prio=9 tid=0x000000014f856800 nid=0x150e04000 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Surrogate Locker Thread (CMS)" daemon prio=5 tid=0x000000014f856000 nid=0x150d01000 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" daemon prio=8 tid=0x000000014f828000 nid=0x150a01000 in Object.wait() [0x0000000150a00000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <0x0000000110122138> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Locked ownable synchronizers: - None "Reference Handler" daemon prio=10 tid=0x000000014f827800 nid=0x14ef04000 in Object.wait() [0x000000014ef03000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x0000000110138eb0> (a java.lang.ref.Reference$Lock) Locked ownable synchronizers: - None "VM Thread" prio=9 tid=0x000000014f820000 nid=0x14ee01000 runnable "Gang worker#0 (Parallel GC Threads)" prio=9 tid=0x0000000102802800 nid=0x101e01000 runnable "Gang worker#1 (Parallel GC Threads)" prio=9 tid=0x0000000102803000 nid=0x101f04000 runnable "Gang worker#2 (Parallel GC Threads)" prio=9 tid=0x0000000102804000 nid=0x106102000 runnable "Gang worker#3 (Parallel GC Threads)" prio=9 tid=0x0000000106800000 nid=0x106205000 runnable "Gang worker#4 (Parallel GC Threads)" prio=9 tid=0x0000000106801000 nid=0x106308000 runnable "Gang worker#5 (Parallel GC Threads)" prio=9 tid=0x0000000106801800 nid=0x10640b000 runnable "Gang worker#6 (Parallel GC Threads)" prio=9 tid=0x0000000106802000 nid=0x10650e000 runnable "Gang worker#7 (Parallel GC Threads)" prio=9 tid=0x0000000106802800 nid=0x106611000 runnable "Concurrent Mark-Sweep GC Thread" prio=9 tid=0x00000001068bd000 nid=0x14e823000 runnable "Gang worker#0 (Parallel CMS Threads)" prio=9 tid=0x00000001068bc000 nid=0x14e61d000 runnable "Gang worker#1 (Parallel CMS Threads)" prio=9 tid=0x00000001068bc800 nid=0x14e720000 runnable "VM Periodic Task Thread" prio=10 tid=0x000000014f859800 nid=0x151c07000 waiting on condition "Exception Catcher Thread" prio=10 tid=0x0000000102800000 nid=0x101b01000 runnable JNI global references: 1121 =====================================
