Would you mind pastebinning the stacktrace? It doesn't looks like https://issues.apache.org/jira/browse/HDFS-88 (HBASE-667) going by the below, an issue that HADOOP-5859 purportedly fixes -- I see you commented on it -- but our Todd thinks otherwise (He has a 'real' fix up in another issue that I currently can't put my finger on). St.Ack
On Fri, Jul 16, 2010 at 7:19 AM, Luke Forehand <[email protected]> wrote: > > I grepped yesterday's logs on all servers for "Blocking updates" and there > was no trace. I believe I had encountered the blocking updates problem > earlier in the project but throttled down the import speed which seemed to > fix that. > > I just double checked and all three region servers were idle. Something > interesting that I noticed however, was that each regionserver had a > particular ResponseProcessor thread running for a specific block, and that > thread was stuck in a running state during the entirety of the hang. Also a > DataStreamer thread for the block associated with the ResponseProcessor was > in a wait state. This makes me think that each server was stuck operating on > a specific block. > > "ResponseProcessor for block blk_1926230463847049982_2694658" - Thread > t...@61160 > java.lang.Thread.State: RUNNABLE > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > - locked sun.nio.ch.uti...@196fbfd0 > - locked java.util.collections$unmodifiable...@7799fdbb > - locked sun.nio.ch.epollselectori...@1ee13d55 > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > at > org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332) > at > org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) > at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) > at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) > at java.io.DataInputStream.readFully(DataInputStream.java:178) > at java.io.DataInputStream.readLong(DataInputStream.java:399) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2399) > > Locked ownable synchronizers: > - None > > "DataStreamer for file > /hbase/.logs/dn01.colo.networkedinsights.com,60020,1279222293084/hlog.dat.1279228611023 > block blk_1926230463847049982_2694658" - Thread t...@61158 > java.lang.Thread.State: TIMED_WAITING on java.util.linkedl...@475b455c > at java.lang.Object.wait(Native Method) > at > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2247) > > Locked ownable synchronizers: > - None > > On 7/15/10 10:17 PM, "Ryan Rawson" <[email protected]> wrote: > > I'm not seeing anything in that logfile, you are seeing compactions > for various regions, but im not seeing flushes (typical during insert > loads) and nothing else. One thing we look to see is a log message > "Blocking updates" which indicates that a particular region has > decided it's holding up to prevent taking too many inserts. > > Like I said, you could be seeing this on a different regionserver, if > all the clients are blocked on 1 regionserver and can't get to the > others then most will look idle and only one will actually show > anything interesting in the log. > > Can you check for this behaviour? > > Also if you want to tweak the config with the values I pasted that should > help. > > On Thu, Jul 15, 2010 at 7:25 PM, Luke Forehand > <[email protected]> wrote: >> It looks like we are going straight from the default config, no expicit >> setting of anything. >> >> On 7/15/10 9:03 PM, "Ryan Rawson" <[email protected]> wrote: >> >> In this case the regionserver isn't actually doing anything - all the >> IPC thread handlers are waiting in their queue handoff thingy (how >> they get socket/work to do). >> >> Something elsewhere perhaps? Check the logs of your jobs, there might >> be something interesting there. >> >> One thing that frequently happens is you overrun 1 regionserver with >> edits and it isnt flushing fast enough, so it pauses updates and all >> clients end up stuck on it. >> >> What was that config again? I use these settings: >> >> <property> >> <name>hbase.hstore.blockingStoreFiles</name> >> <value>15</value> >> </property> >> >> <property> >> <name>dfs.datanode.socket.write.timeout</name> >> <value>0</value> >> </property> >> >> <property> >> <name>hbase.hregion.memstore.block.multiplier</name> >> <value>8</value> >> </property> >> >> perhaps try these ones? >> >> -ryan >> >> On Thu, Jul 15, 2010 at 6:55 PM, Luke Forehand >> <[email protected]> wrote: >>> >>> Yea I started with 12 task max per node, dropped it to 8, and now 4, but >>> still having the issue. Here is the spec for each box: >>> >>> 2x CPU (2.5 GHZ nehalem ep quad core) >>> 24 GB RAM (4gb / region server ) >>> 4x 1tb hard drives >>> >>> >>> On 7/15/10 6:32 PM, "Ryan Rawson" <[email protected]> wrote: >>> >>> Are you sure you dont have any GC issues? >>> >>> Also you are aware that max.tasks.maximum is _per node_ right? How >>> big are these machines? >>> >>> -ryan >>> >>> On Thu, Jul 15, 2010 at 4:30 PM, Jean-Daniel Cryans <[email protected]> >>> wrote: >>>> Nothing particular in that dump, and I'm not aware of any deadlock in >>>> 0.20.3, could we see the region server log? >>>> >>>> Thx >>>> >>>> J-D >>>> >>>> On Thu, Jul 15, 2010 at 4:17 PM, Luke Forehand >>>> <[email protected]> wrote: >>>>> First post evar! >>>>> >>>>> I have a 3 node cluster and have set >>>>> mapred.tasktracker.map.tasks.maximum=4. I am using Cloudera's CDH2 dist >>>>> (hadoop 0.20.1 and hbase 0.20.3). I am importing a 15GB sequence file >>>>> into an HTable instance straight from the Mapper, no reduce phase. At >>>>> random times during the job, the RegionServer JVMs start to completely >>>>> hang. I started up jstatd on the servers and used VisualVM to profile >>>>> the processes. When I encounter the hang, I notice absolutely no >>>>> problems with heap or garbage collection, the region servers just become >>>>> vegetables within minutes of eachother. This fact, along with the >>>>> randomness of the hang, leads me to believe there is a dead lock due to a >>>>> race condition somewhere in the code. I suspect something similar to the >>>>> following issues: >>>>> >>>>> http://issues.apache.org/jira/browse/HADOOP-5859 >>>>> http://issues.apache.org/jira/browse/HBASE-667 >>>>> >>>>> Here is a thread dump of one of the region server procs immediately after >>>>> it went silent. Would anybody that has more knowledge about the inner >>>>> workings of hadoop/hbase IO have the time to help me confirm that this is >>>>> the issue I am experiencing? >>>>> >>>>> 2010-07-15 16:33:26 >>>>> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode): >>>>> >>>>> "RMI TCP Connection(73)-192.168.100.113" - Thread t...@62011 >>>>> java.lang.Thread.State: TIMED_WAITING on >>>>> com.sun.jmx.remote.internal.arraynotificationbuf...@bd5192 >>>>> at java.lang.Object.wait(Native Method) >>>>> at >>>>> com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:417) >>>>> at >>>>> com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209) >>>>> at >>>>> com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258) >>>>> at >>>>> javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227) >>>>> at >>>>> javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225) >>>>> at >>>>> javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1231) >>>>> at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) >>>>> at sun.rmi.transport.Transport$1.run(Transport.java:159) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - locked java.util.concurrent.locks.reentrantlock$nonfairs...@40aa3008 >>>>> >>>>> "RMI TCP Connection(72)-192.168.100.113" - Thread t...@62010 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at java.net.SocketInputStream.socketRead0(Native Method) >>>>> at java.net.SocketInputStream.read(SocketInputStream.java:129) >>>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) >>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:237) >>>>> - locked java.io.bufferedinputstr...@72bda36f >>>>> at java.io.FilterInputStream.read(FilterInputStream.java:66) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - locked java.util.concurrent.locks.reentrantlock$nonfairs...@350743 >>>>> >>>>> "JMX server connection timeout 62009" - Thread t...@62009 >>>>> java.lang.Thread.State: TIMED_WAITING on [...@5eebd2c6 >>>>> at java.lang.Object.wait(Native Method) >>>>> at >>>>> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "ResponseProcessor for block blk_1926230463847049982_2694658" - Thread >>>>> t...@61160 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked sun.nio.ch.uti...@196fbfd0 >>>>> - locked java.util.collections$unmodifiable...@7799fdbb >>>>> - locked sun.nio.ch.epollselectori...@1ee13d55 >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332) >>>>> at >>>>> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) >>>>> at >>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) >>>>> at >>>>> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) >>>>> at java.io.DataInputStream.readFully(DataInputStream.java:178) >>>>> at java.io.DataInputStream.readLong(DataInputStream.java:399) >>>>> at >>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2399) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "DataStreamer for file >>>>> /hbase/.logs/dn01.colo.networkedinsights.com,60020,1279222293084/hlog.dat.1279228611023 >>>>> block blk_1926230463847049982_2694658" - Thread t...@61158 >>>>> java.lang.Thread.State: TIMED_WAITING on java.util.linkedl...@475b455c >>>>> at java.lang.Object.wait(Native Method) >>>>> at >>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2247) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "RMI TCP Connection(71)-192.168.100.113" - Thread t...@53904 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.management.ThreadImpl.dumpThreads0(Native Method) >>>>> at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374) >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> at >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at >>>>> com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167) >>>>> at >>>>> com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96) >>>>> at >>>>> com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33) >>>>> at >>>>> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) >>>>> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) >>>>> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) >>>>> at javax.management.StandardMBean.invoke(StandardMBean.java:391) >>>>> at >>>>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) >>>>> at >>>>> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) >>>>> at >>>>> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1426) >>>>> at >>>>> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) >>>>> at >>>>> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264) >>>>> at >>>>> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1359) >>>>> at >>>>> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) >>>>> at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) >>>>> at sun.rmi.transport.Transport$1.run(Transport.java:159) >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>> at sun.rmi.transport.Transport.serviceCall(Transport.java:155) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - locked java.util.concurrent.locks.reentrantlock$nonfairs...@5581a749 >>>>> >>>>> "JMX server connection timeout 28949" - Thread t...@28949 >>>>> java.lang.Thread.State: TIMED_WAITING on [...@1c6e538d >>>>> at java.lang.Object.wait(Native Method) >>>>> at >>>>> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "RMI TCP Connection(70)-192.168.100.113" - Thread t...@28947 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at java.net.SocketInputStream.socketRead0(Native Method) >>>>> at java.net.SocketInputStream.read(SocketInputStream.java:129) >>>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) >>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:237) >>>>> - locked java.io.bufferedinputstr...@513a56c5 >>>>> at java.io.FilterInputStream.read(FilterInputStream.java:66) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - locked java.util.concurrent.locks.reentrantlock$nonfairs...@4d16c92c >>>>> >>>>> "regionserver/192.168.200.31:60020.compactor-EventThread" - Thread >>>>> t...@19245 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@35c7b446 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:414) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "regionserver/192.168.200.31:60020.compactor-SendThread" - Thread >>>>> t...@19244 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked sun.nio.ch.uti...@30f33929 >>>>> - locked java.util.collections$unmodifiable...@ea26dec >>>>> - locked sun.nio.ch.epollselectori...@3905bf36 >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:921) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "RMI Scheduler(0)" - Thread t...@406 >>>>> java.lang.Thread.State: TIMED_WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@66d9c57f >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >>>>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) >>>>> at >>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) >>>>> at >>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "pool-1-thread-1" - Thread t...@74 >>>>> java.lang.Thread.State: TIMED_WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@3175e1a2 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >>>>> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) >>>>> at >>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) >>>>> at >>>>> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "LruBlockCache.EvictionThread" - Thread t...@72 >>>>> java.lang.Thread.State: WAITING on >>>>> org.apache.hadoop.hbase.io.hfile.lrublockcache$evictionthr...@32306c0c >>>>> at java.lang.Object.wait(Native Method) >>>>> at java.lang.Object.wait(Object.java:485) >>>>> at >>>>> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:512) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 24 on 60020" - Thread t...@71 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 23 on 60020" - Thread t...@70 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 22 on 60020" - Thread t...@69 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 21 on 60020" - Thread t...@68 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 20 on 60020" - Thread t...@67 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 19 on 60020" - Thread t...@66 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 18 on 60020" - Thread t...@65 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 17 on 60020" - Thread t...@64 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 16 on 60020" - Thread t...@63 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 15 on 60020" - Thread t...@62 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 14 on 60020" - Thread t...@61 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 13 on 60020" - Thread t...@60 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 12 on 60020" - Thread t...@59 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 11 on 60020" - Thread t...@58 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 10 on 60020" - Thread t...@57 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 9 on 60020" - Thread t...@56 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 8 on 60020" - Thread t...@55 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 7 on 60020" - Thread t...@54 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 6 on 60020" - Thread t...@53 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 5 on 60020" - Thread t...@52 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 4 on 60020" - Thread t...@51 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 3 on 60020" - Thread t...@50 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 2 on 60020" - Thread t...@49 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 1 on 60020" - Thread t...@48 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server handler 0 on 60020" - Thread t...@47 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server listener on 60020" - Thread t...@14 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked sun.nio.ch.uti...@284c8f79 >>>>> - locked java.util.collections$unmodifiable...@3a5f659a >>>>> - locked sun.nio.ch.epollselectori...@4d077f66 >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Server Responder" - Thread t...@16 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked sun.nio.ch.uti...@273d290c >>>>> - locked java.util.collections$unmodifiable...@2051e6f4 >>>>> - locked sun.nio.ch.epollselectori...@6111adbf >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:477) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "Timer-0" - Thread t...@46 >>>>> java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@50b27cde >>>>> at java.lang.Object.wait(Native Method) >>>>> at java.util.TimerThread.mainLoop(Timer.java:509) >>>>> at java.util.TimerThread.run(Timer.java:462) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "889074...@qtp0-1 - Acceptor0 [email protected]:60030" - >>>>> Thread t...@45 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked sun.nio.ch.uti...@1d7082c8 >>>>> - locked java.util.collections$unmodifiable...@b9bc44a >>>>> - locked sun.nio.ch.epollselectori...@6f7f73f7 >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at >>>>> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429) >>>>> at >>>>> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185) >>>>> at >>>>> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) >>>>> at >>>>> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707) >>>>> at >>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "1374821...@qtp0-0" - Thread t...@44 >>>>> java.lang.Thread.State: TIMED_WAITING on >>>>> org.mortbay.thread.queuedthreadpool$poolthr...@51f21c50 >>>>> at java.lang.Object.wait(Native Method) >>>>> at >>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:565) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "regionserver/192.168.200.31:60020.leaseChecker" - Thread t...@23 >>>>> java.lang.Thread.State: TIMED_WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@7623fc35 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >>>>> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201) >>>>> at org.apache.hadoop.hbase.Leases.run(Leases.java:81) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "regionserver/192.168.200.31:60020.majorCompactionChecker" - Thread >>>>> t...@22 >>>>> java.lang.Thread.State: TIMED_WAITING >>>>> at java.lang.Thread.sleep(Native Method) >>>>> at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) >>>>> at org.apache.hadoop.hbase.Chore.run(Chore.java:76) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "regionserver/192.168.200.31:60020.worker" - Thread t...@17 >>>>> java.lang.Thread.State: TIMED_WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@3483382b >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) >>>>> at >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1425) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "regionserver/192.168.200.31:60020.compactor" - Thread t...@19 >>>>> java.lang.Thread.State: TIMED_WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@12dca9fc >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) >>>>> at >>>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:85) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "regionserver/192.168.200.31:60020.cacheFlusher" - Thread t...@18 >>>>> java.lang.Thread.State: TIMED_WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@551be910 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at >>>>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) >>>>> at >>>>> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:145) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "regionserver/192.168.200.31:60020.logFlusher" - Thread t...@21 >>>>> java.lang.Thread.State: TIMED_WAITING >>>>> at java.lang.Thread.sleep(Native Method) >>>>> at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) >>>>> at org.apache.hadoop.hbase.Chore.run(Chore.java:76) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "regionserver/192.168.200.31:60020.logRoller" - Thread t...@20 >>>>> java.lang.Thread.State: TIMED_WAITING on >>>>> java.util.concurrent.atomic.atomicbool...@1b624517 >>>>> at java.lang.Object.wait(Native Method) >>>>> at >>>>> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:65) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "Timer thread for monitoring jvm" - Thread t...@42 >>>>> java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@63354999 >>>>> at java.lang.Object.wait(Native Method) >>>>> at java.util.TimerThread.mainLoop(Timer.java:509) >>>>> at java.util.TimerThread.run(Timer.java:462) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "Timer thread for monitoring hbase" - Thread t...@41 >>>>> java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@4c966553 >>>>> at java.lang.Object.wait(Native Method) >>>>> at java.util.TimerThread.mainLoop(Timer.java:509) >>>>> at java.util.TimerThread.run(Timer.java:462) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "LeaseChecker" - Thread t...@40 >>>>> java.lang.Thread.State: TIMED_WAITING >>>>> at java.lang.Thread.sleep(Native Method) >>>>> at >>>>> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "IPC Client (47) connection to /192.168.200.30:60000 from an unknown >>>>> user" - Thread t...@28 >>>>> java.lang.Thread.State: TIMED_WAITING on >>>>> org.apache.hadoop.hbase.ipc.hbaseclient$connect...@3ffe5e31 >>>>> at java.lang.Object.wait(Native Method) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:404) >>>>> at >>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:447) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "DestroyJavaVM" - Thread t...@27 >>>>> java.lang.Thread.State: RUNNABLE >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "regionserver/192.168.200.31:60020" - Thread t...@26 >>>>> java.lang.Thread.State: TIMED_WAITING >>>>> at java.lang.Thread.sleep(Native Method) >>>>> at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) >>>>> at >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:594) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "main-EventThread" - Thread t...@25 >>>>> java.lang.Thread.State: WAITING on >>>>> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@54da03c9 >>>>> at sun.misc.Unsafe.park(Native Method) >>>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >>>>> at >>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >>>>> at >>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >>>>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:414) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "main-SendThread" - Thread t...@24 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >>>>> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >>>>> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >>>>> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >>>>> - locked sun.nio.ch.uti...@18969651 >>>>> - locked java.util.collections$unmodifiable...@4666ac73 >>>>> - locked sun.nio.ch.epollselectori...@7ec3d620 >>>>> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >>>>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:921) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "Timer thread for monitoring rpc" - Thread t...@15 >>>>> java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@352eaca4 >>>>> at java.lang.Object.wait(Native Method) >>>>> at java.util.TimerThread.mainLoop(Timer.java:509) >>>>> at java.util.TimerThread.run(Timer.java:462) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "RMI TCP Accept-0" - Thread t...@12 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at java.net.PlainSocketImpl.socketAccept(Native Method) >>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) >>>>> - locked java.net.sockssocketi...@6b1f0fe4 >>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453) >>>>> at java.net.ServerSocket.accept(ServerSocket.java:421) >>>>> at >>>>> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "RMI TCP Accept-10102" - Thread t...@11 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at java.net.PlainSocketImpl.socketAccept(Native Method) >>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) >>>>> - locked java.net.sockssocketi...@444c78ad >>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453) >>>>> at java.net.ServerSocket.accept(ServerSocket.java:421) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "RMI TCP Accept-0" - Thread t...@10 >>>>> java.lang.Thread.State: RUNNABLE >>>>> at java.net.PlainSocketImpl.socketAccept(Native Method) >>>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) >>>>> - locked java.net.sockssocketi...@7886c8e2 >>>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453) >>>>> at java.net.ServerSocket.accept(ServerSocket.java:421) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) >>>>> at >>>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) >>>>> at java.lang.Thread.run(Thread.java:619) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "Signal Dispatcher" - Thread t...@5 >>>>> java.lang.Thread.State: RUNNABLE >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "Finalizer" - Thread t...@3 >>>>> java.lang.Thread.State: WAITING on >>>>> java.lang.ref.referencequeue$l...@b485e43 >>>>> at java.lang.Object.wait(Native Method) >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) >>>>> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) >>>>> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> "Reference Handler" - Thread t...@2 >>>>> java.lang.Thread.State: WAITING on java.lang.ref.reference$l...@3c24867e >>>>> at java.lang.Object.wait(Native Method) >>>>> at java.lang.Object.wait(Object.java:485) >>>>> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) >>>>> >>>>> Locked ownable synchronizers: >>>>> - None >>>>> >>>>> --- >>>>> >>>>> Respectfully Yours, >>>>> Luke Forehand >>>>> Software Engineer >>>>> Networked Insights >>>>> http://www.networkedinsights.com >>>>> >>> >> >
