First post evar! I have a 3 node cluster and have set mapred.tasktracker.map.tasks.maximum=4. I am using Cloudera's CDH2 dist (hadoop 0.20.1 and hbase 0.20.3). I am importing a 15GB sequence file into an HTable instance straight from the Mapper, no reduce phase. At random times during the job, the RegionServer JVMs start to completely hang. I started up jstatd on the servers and used VisualVM to profile the processes. When I encounter the hang, I notice absolutely no problems with heap or garbage collection, the region servers just become vegetables within minutes of eachother. This fact, along with the randomness of the hang, leads me to believe there is a dead lock due to a race condition somewhere in the code. I suspect something similar to the following issues:
http://issues.apache.org/jira/browse/HADOOP-5859 http://issues.apache.org/jira/browse/HBASE-667 Here is a thread dump of one of the region server procs immediately after it went silent. Would anybody that has more knowledge about the inner workings of hadoop/hbase IO have the time to help me confirm that this is the issue I am experiencing? 2010-07-15 16:33:26 Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode): "RMI TCP Connection(73)-192.168.100.113" - Thread t...@62011 java.lang.Thread.State: TIMED_WAITING on com.sun.jmx.remote.internal.arraynotificationbuf...@bd5192 at java.lang.Object.wait(Native Method) at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:417) at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209) at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258) at javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227) at javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225) at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1231) at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - locked java.util.concurrent.locks.reentrantlock$nonfairs...@40aa3008 "RMI TCP Connection(72)-192.168.100.113" - Thread t...@62010 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) - locked java.io.bufferedinputstr...@72bda36f at java.io.FilterInputStream.read(FilterInputStream.java:66) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - locked java.util.concurrent.locks.reentrantlock$nonfairs...@350743 "JMX server connection timeout 62009" - Thread t...@62009 java.lang.Thread.State: TIMED_WAITING on [...@5eebd2c6 at java.lang.Object.wait(Native Method) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None "ResponseProcessor for block blk_1926230463847049982_2694658" - Thread t...@61160 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked sun.nio.ch.uti...@196fbfd0 - locked java.util.collections$unmodifiable...@7799fdbb - locked sun.nio.ch.epollselectori...@1ee13d55 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) at java.io.DataInputStream.readFully(DataInputStream.java:178) at java.io.DataInputStream.readLong(DataInputStream.java:399) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2399) Locked ownable synchronizers: - None "DataStreamer for file /hbase/.logs/dn01.colo.networkedinsights.com,60020,1279222293084/hlog.dat.1279228611023 block blk_1926230463847049982_2694658" - Thread t...@61158 java.lang.Thread.State: TIMED_WAITING on java.util.linkedl...@475b455c at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2247) Locked ownable synchronizers: - None "RMI TCP Connection(71)-192.168.100.113" - Thread t...@53904 java.lang.Thread.State: RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167) at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96) at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at javax.management.StandardMBean.invoke(StandardMBean.java:391) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1426) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1359) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - locked java.util.concurrent.locks.reentrantlock$nonfairs...@5581a749 "JMX server connection timeout 28949" - Thread t...@28949 java.lang.Thread.State: TIMED_WAITING on [...@1c6e538d at java.lang.Object.wait(Native Method) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None "RMI TCP Connection(70)-192.168.100.113" - Thread t...@28947 java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) - locked java.io.bufferedinputstr...@513a56c5 at java.io.FilterInputStream.read(FilterInputStream.java:66) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - locked java.util.concurrent.locks.reentrantlock$nonfairs...@4d16c92c "regionserver/192.168.200.31:60020.compactor-EventThread" - Thread t...@19245 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@35c7b446 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:414) Locked ownable synchronizers: - None "regionserver/192.168.200.31:60020.compactor-SendThread" - Thread t...@19244 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked sun.nio.ch.uti...@30f33929 - locked java.util.collections$unmodifiable...@ea26dec - locked sun.nio.ch.epollselectori...@3905bf36 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:921) Locked ownable synchronizers: - None "RMI Scheduler(0)" - Thread t...@406 java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@66d9c57f at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None "pool-1-thread-1" - Thread t...@74 java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@3175e1a2 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None "LruBlockCache.EvictionThread" - Thread t...@72 java.lang.Thread.State: WAITING on org.apache.hadoop.hbase.io.hfile.lrublockcache$evictionthr...@32306c0c at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:512) Locked ownable synchronizers: - None "IPC Server handler 24 on 60020" - Thread t...@71 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 23 on 60020" - Thread t...@70 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 22 on 60020" - Thread t...@69 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 21 on 60020" - Thread t...@68 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 20 on 60020" - Thread t...@67 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 19 on 60020" - Thread t...@66 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 18 on 60020" - Thread t...@65 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 17 on 60020" - Thread t...@64 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 16 on 60020" - Thread t...@63 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 15 on 60020" - Thread t...@62 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 14 on 60020" - Thread t...@61 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 13 on 60020" - Thread t...@60 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 12 on 60020" - Thread t...@59 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 11 on 60020" - Thread t...@58 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 10 on 60020" - Thread t...@57 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 9 on 60020" - Thread t...@56 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 8 on 60020" - Thread t...@55 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 7 on 60020" - Thread t...@54 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 6 on 60020" - Thread t...@53 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 5 on 60020" - Thread t...@52 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 4 on 60020" - Thread t...@51 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 3 on 60020" - Thread t...@50 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 2 on 60020" - Thread t...@49 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 1 on 60020" - Thread t...@48 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server handler 0 on 60020" - Thread t...@47 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) Locked ownable synchronizers: - None "IPC Server listener on 60020" - Thread t...@14 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked sun.nio.ch.uti...@284c8f79 - locked java.util.collections$unmodifiable...@3a5f659a - locked sun.nio.ch.epollselectori...@4d077f66 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308) Locked ownable synchronizers: - None "IPC Server Responder" - Thread t...@16 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked sun.nio.ch.uti...@273d290c - locked java.util.collections$unmodifiable...@2051e6f4 - locked sun.nio.ch.epollselectori...@6111adbf at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:477) Locked ownable synchronizers: - None "Timer-0" - Thread t...@46 java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@50b27cde at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "889074...@qtp0-1 - Acceptor0 [email protected]:60030" - Thread t...@45 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked sun.nio.ch.uti...@1d7082c8 - locked java.util.collections$unmodifiable...@b9bc44a - locked sun.nio.ch.epollselectori...@6f7f73f7 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) Locked ownable synchronizers: - None "1374821...@qtp0-0" - Thread t...@44 java.lang.Thread.State: TIMED_WAITING on org.mortbay.thread.queuedthreadpool$poolthr...@51f21c50 at java.lang.Object.wait(Native Method) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:565) Locked ownable synchronizers: - None "regionserver/192.168.200.31:60020.leaseChecker" - Thread t...@23 java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@7623fc35 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201) at org.apache.hadoop.hbase.Leases.run(Leases.java:81) Locked ownable synchronizers: - None "regionserver/192.168.200.31:60020.majorCompactionChecker" - Thread t...@22 java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) at org.apache.hadoop.hbase.Chore.run(Chore.java:76) Locked ownable synchronizers: - None "regionserver/192.168.200.31:60020.worker" - Thread t...@17 java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@3483382b at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1425) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None "regionserver/192.168.200.31:60020.compactor" - Thread t...@19 java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@12dca9fc at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:85) Locked ownable synchronizers: - None "regionserver/192.168.200.31:60020.cacheFlusher" - Thread t...@18 java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@551be910 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:145) Locked ownable synchronizers: - None "regionserver/192.168.200.31:60020.logFlusher" - Thread t...@21 java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) at org.apache.hadoop.hbase.Chore.run(Chore.java:76) Locked ownable synchronizers: - None "regionserver/192.168.200.31:60020.logRoller" - Thread t...@20 java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.atomic.atomicbool...@1b624517 at java.lang.Object.wait(Native Method) at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:65) Locked ownable synchronizers: - None "Timer thread for monitoring jvm" - Thread t...@42 java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@63354999 at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "Timer thread for monitoring hbase" - Thread t...@41 java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@4c966553 at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "LeaseChecker" - Thread t...@40 java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None "IPC Client (47) connection to /192.168.200.30:60000 from an unknown user" - Thread t...@28 java.lang.Thread.State: TIMED_WAITING on org.apache.hadoop.hbase.ipc.hbaseclient$connect...@3ffe5e31 at java.lang.Object.wait(Native Method) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:404) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:447) Locked ownable synchronizers: - None "DestroyJavaVM" - Thread t...@27 java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "regionserver/192.168.200.31:60020" - Thread t...@26 java.lang.Thread.State: TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:594) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None "main-EventThread" - Thread t...@25 java.lang.Thread.State: WAITING on java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@54da03c9 at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:414) Locked ownable synchronizers: - None "main-SendThread" - Thread t...@24 java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked sun.nio.ch.uti...@18969651 - locked java.util.collections$unmodifiable...@4666ac73 - locked sun.nio.ch.epollselectori...@7ec3d620 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:921) Locked ownable synchronizers: - None "Timer thread for monitoring rpc" - Thread t...@15 java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@352eaca4 at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:509) at java.util.TimerThread.run(Timer.java:462) Locked ownable synchronizers: - None "RMI TCP Accept-0" - Thread t...@12 java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) - locked java.net.sockssocketi...@6b1f0fe4 at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None "RMI TCP Accept-10102" - Thread t...@11 java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) - locked java.net.sockssocketi...@444c78ad at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None "RMI TCP Accept-0" - Thread t...@10 java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) - locked java.net.sockssocketi...@7886c8e2 at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) at java.lang.Thread.run(Thread.java:619) Locked ownable synchronizers: - None "Signal Dispatcher" - Thread t...@5 java.lang.Thread.State: RUNNABLE Locked ownable synchronizers: - None "Finalizer" - Thread t...@3 java.lang.Thread.State: WAITING on java.lang.ref.referencequeue$l...@b485e43 at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Locked ownable synchronizers: - None "Reference Handler" - Thread t...@2 java.lang.Thread.State: WAITING on java.lang.ref.reference$l...@3c24867e at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) Locked ownable synchronizers: - None --- Respectfully Yours, Luke Forehand Software Engineer Networked Insights http://www.networkedinsights.com
