Yea I started with 12 task max per node, dropped it to 8, and now 4, but still having the issue. Here is the spec for each box:
2x CPU (2.5 GHZ nehalem ep quad core) 24 GB RAM (4gb / region server ) 4x 1tb hard drives On 7/15/10 6:32 PM, "Ryan Rawson" <[email protected]> wrote: Are you sure you dont have any GC issues? Also you are aware that max.tasks.maximum is _per node_ right? How big are these machines? -ryan On Thu, Jul 15, 2010 at 4:30 PM, Jean-Daniel Cryans <[email protected]> wrote: > Nothing particular in that dump, and I'm not aware of any deadlock in > 0.20.3, could we see the region server log? > > Thx > > J-D > > On Thu, Jul 15, 2010 at 4:17 PM, Luke Forehand > <[email protected]> wrote: >> First post evar! >> >> I have a 3 node cluster and have set mapred.tasktracker.map.tasks.maximum=4. >> I am using Cloudera's CDH2 dist (hadoop 0.20.1 and hbase 0.20.3). I am >> importing a 15GB sequence file into an HTable instance straight from the >> Mapper, no reduce phase. At random times during the job, the RegionServer >> JVMs start to completely hang. I started up jstatd on the servers and used >> VisualVM to profile the processes. When I encounter the hang, I notice >> absolutely no problems with heap or garbage collection, the region servers >> just become vegetables within minutes of eachother. This fact, along with >> the randomness of the hang, leads me to believe there is a dead lock due to >> a race condition somewhere in the code. I suspect something similar to the >> following issues: >> >> http://issues.apache.org/jira/browse/HADOOP-5859 >> http://issues.apache.org/jira/browse/HBASE-667 >> >> Here is a thread dump of one of the region server procs immediately after it >> went silent. Would anybody that has more knowledge about the inner workings >> of hadoop/hbase IO have the time to help me confirm that this is the issue I >> am experiencing? >> >> 2010-07-15 16:33:26 >> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode): >> >> "RMI TCP Connection(73)-192.168.100.113" - Thread t...@62011 >> java.lang.Thread.State: TIMED_WAITING on >> com.sun.jmx.remote.internal.arraynotificationbuf...@bd5192 >> at java.lang.Object.wait(Native Method) >> at >> com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(ArrayNotificationBuffer.java:417) >> at >> com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(ArrayNotificationBuffer.java:209) >> at >> com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(ServerNotifForwarder.java:258) >> at >> javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1227) >> at >> javax.management.remote.rmi.RMIConnectionImpl$2.run(RMIConnectionImpl.java:1225) >> at >> javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(RMIConnectionImpl.java:1231) >> at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) >> at sun.rmi.transport.Transport$1.run(Transport.java:159) >> at java.security.AccessController.doPrivileged(Native Method) >> at sun.rmi.transport.Transport.serviceCall(Transport.java:155) >> at >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - locked java.util.concurrent.locks.reentrantlock$nonfairs...@40aa3008 >> >> "RMI TCP Connection(72)-192.168.100.113" - Thread t...@62010 >> java.lang.Thread.State: RUNNABLE >> at java.net.SocketInputStream.socketRead0(Native Method) >> at java.net.SocketInputStream.read(SocketInputStream.java:129) >> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) >> at java.io.BufferedInputStream.read(BufferedInputStream.java:237) >> - locked java.io.bufferedinputstr...@72bda36f >> at java.io.FilterInputStream.read(FilterInputStream.java:66) >> at >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - locked java.util.concurrent.locks.reentrantlock$nonfairs...@350743 >> >> "JMX server connection timeout 62009" - Thread t...@62009 >> java.lang.Thread.State: TIMED_WAITING on [...@5eebd2c6 >> at java.lang.Object.wait(Native Method) >> at >> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - None >> >> "ResponseProcessor for block blk_1926230463847049982_2694658" - Thread >> t...@61160 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >> - locked sun.nio.ch.uti...@196fbfd0 >> - locked java.util.collections$unmodifiable...@7799fdbb >> - locked sun.nio.ch.epollselectori...@1ee13d55 >> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >> at >> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332) >> at >> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) >> at >> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) >> at >> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) >> at java.io.DataInputStream.readFully(DataInputStream.java:178) >> at java.io.DataInputStream.readLong(DataInputStream.java:399) >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2399) >> >> Locked ownable synchronizers: >> - None >> >> "DataStreamer for file >> /hbase/.logs/dn01.colo.networkedinsights.com,60020,1279222293084/hlog.dat.1279228611023 >> block blk_1926230463847049982_2694658" - Thread t...@61158 >> java.lang.Thread.State: TIMED_WAITING on java.util.linkedl...@475b455c >> at java.lang.Object.wait(Native Method) >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2247) >> >> Locked ownable synchronizers: >> - None >> >> "RMI TCP Connection(71)-192.168.100.113" - Thread t...@53904 >> java.lang.Thread.State: RUNNABLE >> at sun.management.ThreadImpl.dumpThreads0(Native Method) >> at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:167) >> at >> com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:96) >> at >> com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:33) >> at >> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) >> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) >> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) >> at javax.management.StandardMBean.invoke(StandardMBean.java:391) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) >> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) >> at >> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1426) >> at >> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) >> at >> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1264) >> at >> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1359) >> at >> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) >> at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) >> at sun.rmi.transport.Transport$1.run(Transport.java:159) >> at java.security.AccessController.doPrivileged(Native Method) >> at sun.rmi.transport.Transport.serviceCall(Transport.java:155) >> at >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - locked java.util.concurrent.locks.reentrantlock$nonfairs...@5581a749 >> >> "JMX server connection timeout 28949" - Thread t...@28949 >> java.lang.Thread.State: TIMED_WAITING on [...@1c6e538d >> at java.lang.Object.wait(Native Method) >> at >> com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - None >> >> "RMI TCP Connection(70)-192.168.100.113" - Thread t...@28947 >> java.lang.Thread.State: RUNNABLE >> at java.net.SocketInputStream.socketRead0(Native Method) >> at java.net.SocketInputStream.read(SocketInputStream.java:129) >> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) >> at java.io.BufferedInputStream.read(BufferedInputStream.java:237) >> - locked java.io.bufferedinputstr...@513a56c5 >> at java.io.FilterInputStream.read(FilterInputStream.java:66) >> at >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - locked java.util.concurrent.locks.reentrantlock$nonfairs...@4d16c92c >> >> "regionserver/192.168.200.31:60020.compactor-EventThread" - Thread t...@19245 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@35c7b446 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:414) >> >> Locked ownable synchronizers: >> - None >> >> "regionserver/192.168.200.31:60020.compactor-SendThread" - Thread t...@19244 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >> - locked sun.nio.ch.uti...@30f33929 >> - locked java.util.collections$unmodifiable...@ea26dec >> - locked sun.nio.ch.epollselectori...@3905bf36 >> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:921) >> >> Locked ownable synchronizers: >> - None >> >> "RMI Scheduler(0)" - Thread t...@406 >> java.lang.Thread.State: TIMED_WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@66d9c57f >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) >> at >> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - None >> >> "pool-1-thread-1" - Thread t...@74 >> java.lang.Thread.State: TIMED_WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@3175e1a2 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) >> at >> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - None >> >> "LruBlockCache.EvictionThread" - Thread t...@72 >> java.lang.Thread.State: WAITING on >> org.apache.hadoop.hbase.io.hfile.lrublockcache$evictionthr...@32306c0c >> at java.lang.Object.wait(Native Method) >> at java.lang.Object.wait(Object.java:485) >> at >> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:512) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 24 on 60020" - Thread t...@71 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 23 on 60020" - Thread t...@70 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 22 on 60020" - Thread t...@69 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 21 on 60020" - Thread t...@68 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 20 on 60020" - Thread t...@67 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 19 on 60020" - Thread t...@66 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 18 on 60020" - Thread t...@65 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 17 on 60020" - Thread t...@64 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 16 on 60020" - Thread t...@63 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 15 on 60020" - Thread t...@62 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 14 on 60020" - Thread t...@61 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 13 on 60020" - Thread t...@60 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 12 on 60020" - Thread t...@59 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 11 on 60020" - Thread t...@58 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 10 on 60020" - Thread t...@57 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 9 on 60020" - Thread t...@56 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 8 on 60020" - Thread t...@55 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 7 on 60020" - Thread t...@54 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 6 on 60020" - Thread t...@53 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 5 on 60020" - Thread t...@52 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 4 on 60020" - Thread t...@51 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 3 on 60020" - Thread t...@50 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 2 on 60020" - Thread t...@49 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 1 on 60020" - Thread t...@48 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server handler 0 on 60020" - Thread t...@47 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@905a192 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server listener on 60020" - Thread t...@14 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >> - locked sun.nio.ch.uti...@284c8f79 >> - locked java.util.collections$unmodifiable...@3a5f659a >> - locked sun.nio.ch.epollselectori...@4d077f66 >> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Server Responder" - Thread t...@16 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >> - locked sun.nio.ch.uti...@273d290c >> - locked java.util.collections$unmodifiable...@2051e6f4 >> - locked sun.nio.ch.epollselectori...@6111adbf >> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:477) >> >> Locked ownable synchronizers: >> - None >> >> "Timer-0" - Thread t...@46 >> java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@50b27cde >> at java.lang.Object.wait(Native Method) >> at java.util.TimerThread.mainLoop(Timer.java:509) >> at java.util.TimerThread.run(Timer.java:462) >> >> Locked ownable synchronizers: >> - None >> >> "889074...@qtp0-1 - Acceptor0 [email protected]:60030" - Thread >> t...@45 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >> - locked sun.nio.ch.uti...@1d7082c8 >> - locked java.util.collections$unmodifiable...@b9bc44a >> - locked sun.nio.ch.epollselectori...@6f7f73f7 >> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >> at >> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429) >> at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185) >> at >> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) >> at >> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707) >> at >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) >> >> Locked ownable synchronizers: >> - None >> >> "1374821...@qtp0-0" - Thread t...@44 >> java.lang.Thread.State: TIMED_WAITING on >> org.mortbay.thread.queuedthreadpool$poolthr...@51f21c50 >> at java.lang.Object.wait(Native Method) >> at >> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:565) >> >> Locked ownable synchronizers: >> - None >> >> "regionserver/192.168.200.31:60020.leaseChecker" - Thread t...@23 >> java.lang.Thread.State: TIMED_WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@7623fc35 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >> at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201) >> at org.apache.hadoop.hbase.Leases.run(Leases.java:81) >> >> Locked ownable synchronizers: >> - None >> >> "regionserver/192.168.200.31:60020.majorCompactionChecker" - Thread t...@22 >> java.lang.Thread.State: TIMED_WAITING >> at java.lang.Thread.sleep(Native Method) >> at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) >> at org.apache.hadoop.hbase.Chore.run(Chore.java:76) >> >> Locked ownable synchronizers: >> - None >> >> "regionserver/192.168.200.31:60020.worker" - Thread t...@17 >> java.lang.Thread.State: TIMED_WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@3483382b >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >> at >> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1425) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - None >> >> "regionserver/192.168.200.31:60020.compactor" - Thread t...@19 >> java.lang.Thread.State: TIMED_WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@12dca9fc >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >> at >> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) >> at >> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:85) >> >> Locked ownable synchronizers: >> - None >> >> "regionserver/192.168.200.31:60020.cacheFlusher" - Thread t...@18 >> java.lang.Thread.State: TIMED_WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@551be910 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) >> at >> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) >> at >> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:145) >> >> Locked ownable synchronizers: >> - None >> >> "regionserver/192.168.200.31:60020.logFlusher" - Thread t...@21 >> java.lang.Thread.State: TIMED_WAITING >> at java.lang.Thread.sleep(Native Method) >> at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) >> at org.apache.hadoop.hbase.Chore.run(Chore.java:76) >> >> Locked ownable synchronizers: >> - None >> >> "regionserver/192.168.200.31:60020.logRoller" - Thread t...@20 >> java.lang.Thread.State: TIMED_WAITING on >> java.util.concurrent.atomic.atomicbool...@1b624517 >> at java.lang.Object.wait(Native Method) >> at org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:65) >> >> Locked ownable synchronizers: >> - None >> >> "Timer thread for monitoring jvm" - Thread t...@42 >> java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@63354999 >> at java.lang.Object.wait(Native Method) >> at java.util.TimerThread.mainLoop(Timer.java:509) >> at java.util.TimerThread.run(Timer.java:462) >> >> Locked ownable synchronizers: >> - None >> >> "Timer thread for monitoring hbase" - Thread t...@41 >> java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@4c966553 >> at java.lang.Object.wait(Native Method) >> at java.util.TimerThread.mainLoop(Timer.java:509) >> at java.util.TimerThread.run(Timer.java:462) >> >> Locked ownable synchronizers: >> - None >> >> "LeaseChecker" - Thread t...@40 >> java.lang.Thread.State: TIMED_WAITING >> at java.lang.Thread.sleep(Native Method) >> at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1066) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - None >> >> "IPC Client (47) connection to /192.168.200.30:60000 from an unknown user" - >> Thread t...@28 >> java.lang.Thread.State: TIMED_WAITING on >> org.apache.hadoop.hbase.ipc.hbaseclient$connect...@3ffe5e31 >> at java.lang.Object.wait(Native Method) >> at >> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:404) >> at >> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:447) >> >> Locked ownable synchronizers: >> - None >> >> "DestroyJavaVM" - Thread t...@27 >> java.lang.Thread.State: RUNNABLE >> >> Locked ownable synchronizers: >> - None >> >> "regionserver/192.168.200.31:60020" - Thread t...@26 >> java.lang.Thread.State: TIMED_WAITING >> at java.lang.Thread.sleep(Native Method) >> at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:594) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - None >> >> "main-EventThread" - Thread t...@25 >> java.lang.Thread.State: WAITING on >> java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobj...@54da03c9 >> at sun.misc.Unsafe.park(Native Method) >> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) >> at >> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) >> at >> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) >> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:414) >> >> Locked ownable synchronizers: >> - None >> >> "main-SendThread" - Thread t...@24 >> java.lang.Thread.State: RUNNABLE >> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) >> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215) >> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) >> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) >> - locked sun.nio.ch.uti...@18969651 >> - locked java.util.collections$unmodifiable...@4666ac73 >> - locked sun.nio.ch.epollselectori...@7ec3d620 >> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) >> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:921) >> >> Locked ownable synchronizers: >> - None >> >> "Timer thread for monitoring rpc" - Thread t...@15 >> java.lang.Thread.State: TIMED_WAITING on java.util.taskqu...@352eaca4 >> at java.lang.Object.wait(Native Method) >> at java.util.TimerThread.mainLoop(Timer.java:509) >> at java.util.TimerThread.run(Timer.java:462) >> >> Locked ownable synchronizers: >> - None >> >> "RMI TCP Accept-0" - Thread t...@12 >> java.lang.Thread.State: RUNNABLE >> at java.net.PlainSocketImpl.socketAccept(Native Method) >> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) >> - locked java.net.sockssocketi...@6b1f0fe4 >> at java.net.ServerSocket.implAccept(ServerSocket.java:453) >> at java.net.ServerSocket.accept(ServerSocket.java:421) >> at >> sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34) >> at >> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) >> at >> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - None >> >> "RMI TCP Accept-10102" - Thread t...@11 >> java.lang.Thread.State: RUNNABLE >> at java.net.PlainSocketImpl.socketAccept(Native Method) >> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) >> - locked java.net.sockssocketi...@444c78ad >> at java.net.ServerSocket.implAccept(ServerSocket.java:453) >> at java.net.ServerSocket.accept(ServerSocket.java:421) >> at >> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) >> at >> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - None >> >> "RMI TCP Accept-0" - Thread t...@10 >> java.lang.Thread.State: RUNNABLE >> at java.net.PlainSocketImpl.socketAccept(Native Method) >> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) >> - locked java.net.sockssocketi...@7886c8e2 >> at java.net.ServerSocket.implAccept(ServerSocket.java:453) >> at java.net.ServerSocket.accept(ServerSocket.java:421) >> at >> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) >> at >> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) >> at java.lang.Thread.run(Thread.java:619) >> >> Locked ownable synchronizers: >> - None >> >> "Signal Dispatcher" - Thread t...@5 >> java.lang.Thread.State: RUNNABLE >> >> Locked ownable synchronizers: >> - None >> >> "Finalizer" - Thread t...@3 >> java.lang.Thread.State: WAITING on >> java.lang.ref.referencequeue$l...@b485e43 >> at java.lang.Object.wait(Native Method) >> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) >> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) >> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) >> >> Locked ownable synchronizers: >> - None >> >> "Reference Handler" - Thread t...@2 >> java.lang.Thread.State: WAITING on java.lang.ref.reference$l...@3c24867e >> at java.lang.Object.wait(Native Method) >> at java.lang.Object.wait(Object.java:485) >> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) >> >> Locked ownable synchronizers: >> - None >> >> --- >> >> Respectfully Yours, >> Luke Forehand >> Software Engineer >> Networked Insights >> http://www.networkedinsights.com >>
