Hey Ben, Strange you didn't get the attachment, my gmail is showing the paper clip thingy for that message. ANyway, I have pasted the whole jstack output into this email, since it's pretty small.
Satish 2009-06-02 11:56:26 Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.6.0_03-b05 mixed mode): "Attach Listener" daemon prio=10 tid=0x0000000043e71800 nid=0x5566 waiting on condition [0x0000000000000000..0x0000000000000000] java.lang.Thread.State: RUNNABLE "SyncThread:4" prio=10 tid=0x00002aaac8274800 nid=0x1ced waiting on condition [0x0000000042d63000..0x0000000042d63c10] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaab34e44d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:71) "FollowerRequestProcessor:4" prio=10 tid=0x00002aaac8273c00 nid=0x1cec waiting on condition [0x0000000042b61000..0x0000000042b61b90] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaab34e4648> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:58) "CommitProcessor:4" prio=10 tid=0x00002aaac8150400 nid=0x1ceb in Object.wait() [0x0000000042c62000..0x0000000042c62b10] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaab34e46f0> (a org.apache.zookeeper.server.quorum.CommitProcessor) at java.lang.Object.wait(Object.java:485) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:80) - locked <0x00002aaab34e46f0> (a org.apache.zookeeper.server.quorum.CommitProcessor) "Thread-13" daemon prio=10 tid=0x00002aaac80f9800 nid=0x1b5e runnable [0x0000000042a60000..0x0000000042a60c90] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:206) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) - locked <0x00002aaab34bf5c8> (a java.lang.Object) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:551) "Thread-12" daemon prio=10 tid=0x00002aaac80f8c00 nid=0x1b5d waiting on condition [0x0000000042258000..0x0000000042258c10] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaab34bf7c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:479) "Thread-11" prio=10 tid=0x0000000043d22800 nid=0x1b5c runnable [0x000000004295f000..0x000000004295fb90] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:206) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) - locked <0x00002aaab34bf9a8> (a java.lang.Object) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:551) "Thread-10" prio=10 tid=0x0000000043d21400 nid=0x1b5b waiting on condition [0x000000004285e000..0x000000004285eb10] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaab34bfbf8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:479) "Thread-9" prio=10 tid=0x0000000043d1d800 nid=0x1b5a runnable [0x000000004275d000..0x000000004275da90] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:206) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) - locked <0x00002aaab34c06c8> (a java.lang.Object) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:551) "Thread-8" prio=10 tid=0x0000000043c01800 nid=0x1b59 waiting on condition [0x0000000042359000..0x0000000042359a10] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaab34bf7c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:479) "Thread-7" daemon prio=10 tid=0x00002aaac80f7800 nid=0x1b58 runnable [0x000000004265c000..0x000000004265cd90] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:206) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) - locked <0x00002aaab34c4d98> (a java.lang.Object) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:551) "Thread-6" daemon prio=10 tid=0x00002aaac80f5000 nid=0x1b57 waiting on condition [0x000000004255b000..0x000000004255bd10] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaab34c44b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:479) "Thread-5" daemon prio=10 tid=0x00002aaac80f3800 nid=0x1b56 runnable [0x000000004245a000..0x000000004245ac90] java.lang.Thread.State: RUNNABLE at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) at sun.nio.ch.IOUtil.read(IOUtil.java:206) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) - locked <0x00002aaab34bf440> (a java.lang.Object) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:551) "Thread-2" daemon prio=10 tid=0x0000000043c00400 nid=0x1b53 waiting on condition [0x0000000042157000..0x0000000042157b10] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaab34c5438> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:479) "QuorumPeer:/0:0:0:0:0:0:0:0:2181" prio=10 tid=0x00002aaac80f2400 nid=0x1b4f runnable [0x0000000042056000..0x0000000042056a90] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:237) - locked <0x00002aaab34da038> (a java.io.BufferedInputStream) at java.io.DataInputStream.readInt(DataInputStream.java:370) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:64) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:65) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:109) at org.apache.zookeeper.server.quorum.Follower.readPacket(Follower.java:109) - locked <0x00002aaab34d9fe8> (a org.apache.jute.BinaryInputArchive) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:221) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:450) "WorkerReceiver Thread" daemon prio=10 tid=0x00002aaac80f1800 nid=0x1b4e waiting on condition [0x0000000041f55000..0x0000000041f55a10] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaab34be738> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerReceiver.run(FastLeaderElection.java:186) at java.lang.Thread.run(Thread.java:619) "WorkerSender Thread" daemon prio=10 tid=0x00002aaac80f1000 nid=0x1b4d waiting on condition [0x0000000041ae5000..0x0000000041ae5d90] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaab34be868> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:287) at java.lang.Thread.run(Thread.java:619) "Thread-1" prio=10 tid=0x00002aaac80ec800 nid=0x1b4a runnable [0x00000000419e4000..0x00000000419e4d10] java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145) - locked <0x00002aaab34bcf70> (a java.lang.Object) at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:413) "NIOServerCxn.Factory:2181" daemon prio=10 tid=0x00002aaac8066c00 nid=0x1af6 runnable [0x0000000041333000..0x0000000041333c90] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:184) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x00002aaab3489ae8> (a sun.nio.ch.Util$1) - locked <0x00002aaab3489ad0> (a java.util.Collections$UnmodifiableSet) - locked <0x00002aaab34893f8> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:142) "RMI TCP Accept-0" daemon prio=10 tid=0x00002aaac805d800 nid=0x1af4 runnable [0x0000000041131000..0x0000000041131b90] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) - locked <0x00002aaab3488238> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) at java.lang.Thread.run(Thread.java:619) "Low Memory Detector" daemon prio=10 tid=0x000000004398dc00 nid=0x1af3 runnable [0x0000000000000000..0x0000000000000000] java.lang.Thread.State: RUNNABLE "CompilerThread1" daemon prio=10 tid=0x000000004398b400 nid=0x1af2 waiting on condition [0x0000000000000000..0x0000000040f2ea80] java.lang.Thread.State: RUNNABLE "CompilerThread0" daemon prio=10 tid=0x0000000043989800 nid=0x1af1 waiting on condition [0x0000000000000000..0x0000000040e2da10] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x0000000043988000 nid=0x1af0 runnable [0x0000000000000000..0x0000000040d2da20] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x0000000043964c00 nid=0x1aef in Object.wait() [0x0000000041e54000..0x0000000041e54d10] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaab34b1a18> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) - locked <0x00002aaab34b1a18> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=10 tid=0x0000000043964000 nid=0x1aee in Object.wait() [0x0000000041d53000..0x0000000041d53c90] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaab349abe0> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x00002aaab349abe0> (a java.lang.ref.Reference$Lock) "main" prio=10 tid=0x00000000438ae800 nid=0x1ae4 in Object.wait() [0x0000000040406000..0x0000000040406ed0] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaab3488a38> (a org.apache.zookeeper.server.quorum.QuorumPeer) at java.lang.Thread.join(Thread.java:1143) - locked <0x00002aaab3488a38> (a org.apache.zookeeper.server.quorum.QuorumPeer) at java.lang.Thread.join(Thread.java:1196) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runPeer(QuorumPeerMain.java:116) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:86) "VM Thread" prio=10 tid=0x000000004395f400 nid=0x1aed runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000438b9000 nid=0x1ae5 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000438ba000 nid=0x1ae6 runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x00000000438bb400 nid=0x1ae7 runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x00000000438bc800 nid=0x1ae8 runnable "GC task thread#4 (ParallelGC)" prio=10 tid=0x00000000438bdc00 nid=0x1ae9 runnable "GC task thread#5 (ParallelGC)" prio=10 tid=0x00000000438bf000 nid=0x1aea runnable "GC task thread#6 (ParallelGC)" prio=10 tid=0x00000000438c0400 nid=0x1aeb runnable "GC task thread#7 (ParallelGC)" prio=10 tid=0x00000000438c1800 nid=0x1aec runnable "VM Periodic Task Thread" prio=10 tid=0x00002aaac8065800 nid=0x1af5 waiting on condition JNI global references: 895 On Tue, Jun 2, 2009 at 1:22 PM, Benjamin Reed <br...@yahoo-inc.com> wrote: > can you attach the jstack output? it seems to be missing from your email. > > ben > > > Satish Bhatti wrote: > >> I am running a 5 node ZooKeeper cluster and I noticed that one of them has >> very high CPU usage: >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 6883 infact 22 0 725m 41m 4188 S 95 0.5 >> 5671:54 java >> >> It is not "doing anything" application-wise at this point, so I was >> wondering why the heck it's using up so much CPU! I have attached a jstack >> logfile to this email. >> >> Satish >> >> >
