Created a Jira and attached logile + jstack file to it. https://issues.apache.org/jira/secure/ManageAttachments.jspa?id=12426974
On Tue, Jun 2, 2009 at 1:50 PM, Mahadev Konar <maha...@yahoo-inc.com> wrote: > Hi Satish, > Can you attach this trace to a jira? Please open one for this. Also, can > you do the following - > > For all the threads for the zookeeper server you are seeing the problem on, > > Can you do an strace on all the threads and see which thread is spinning? > > Also, can you upload the configs of the servers to the jira that you > create? > > mahadev > > > On 6/2/09 1:29 PM, "Satish Bhatti" <cthd2...@gmail.com> wrote: > > > Hey Ben, > > Strange you didn't get the attachment, my gmail is showing the paper clip > > thingy for that message. ANyway, I have pasted the whole jstack output > into > > this email, since it's pretty small. > > > > Satish > > > > 2009-06-02 11:56:26 > > Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.6.0_03-b05 mixed > > mode): > > > > "Attach Listener" daemon prio=10 tid=0x0000000043e71800 nid=0x5566 > waiting > > on condition [0x0000000000000000..0x0000000000000000] > > java.lang.Thread.State: RUNNABLE > > > > "SyncThread:4" prio=10 tid=0x00002aaac8274800 nid=0x1ced waiting on > > condition [0x0000000042d63000..0x0000000042d63c10] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab34e44d8> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Ab > > stractQueuedSynchronizer.java:1925) > > at > > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > > at > > > org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java > > :71) > > > > "FollowerRequestProcessor:4" prio=10 tid=0x00002aaac8273c00 nid=0x1cec > > waiting on condition [0x0000000042b61000..0x0000000042b61b90] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab34e4648> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Ab > > stractQueuedSynchronizer.java:1925) > > at > > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > > at > > > org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerReques > > tProcessor.java:58) > > > > "CommitProcessor:4" prio=10 tid=0x00002aaac8150400 nid=0x1ceb in > > Object.wait() [0x0000000042c62000..0x0000000042c62b10] > > java.lang.Thread.State: WAITING (on object monitor) > > at java.lang.Object.wait(Native Method) > > - waiting on <0x00002aaab34e46f0> (a > > org.apache.zookeeper.server.quorum.CommitProcessor) > > at java.lang.Object.wait(Object.java:485) > > at > > > > org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:80> > ) > > - locked <0x00002aaab34e46f0> (a > > org.apache.zookeeper.server.quorum.CommitProcessor) > > > > "Thread-13" daemon prio=10 tid=0x00002aaac80f9800 nid=0x1b5e runnable > > [0x0000000042a60000..0x0000000042a60c90] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.FileDispatcher.read0(Native Method) > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) > > at sun.nio.ch.IOUtil.read(IOUtil.java:206) > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) > > - locked <0x00002aaab34bf5c8> (a java.lang.Object) > > at > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxMa > > nager.java:551) > > > > "Thread-12" daemon prio=10 tid=0x00002aaac80f8c00 nid=0x1b5d waiting on > > condition [0x0000000042258000..0x0000000042258c10] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab34bf7c0> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Ab > > stractQueuedSynchronizer.java:1925) > > at > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) > > at > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxMa > > nager.java:479) > > > > "Thread-11" prio=10 tid=0x0000000043d22800 nid=0x1b5c runnable > > [0x000000004295f000..0x000000004295fb90] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.FileDispatcher.read0(Native Method) > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) > > at sun.nio.ch.IOUtil.read(IOUtil.java:206) > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) > > - locked <0x00002aaab34bf9a8> (a java.lang.Object) > > at > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxMa > > nager.java:551) > > > > "Thread-10" prio=10 tid=0x0000000043d21400 nid=0x1b5b waiting on > condition > > [0x000000004285e000..0x000000004285eb10] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab34bfbf8> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Ab > > stractQueuedSynchronizer.java:1925) > > at > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) > > at > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxMa > > nager.java:479) > > > > "Thread-9" prio=10 tid=0x0000000043d1d800 nid=0x1b5a runnable > > [0x000000004275d000..0x000000004275da90] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.FileDispatcher.read0(Native Method) > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) > > at sun.nio.ch.IOUtil.read(IOUtil.java:206) > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) > > - locked <0x00002aaab34c06c8> (a java.lang.Object) > > at > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxMa > > nager.java:551) > > > > "Thread-8" prio=10 tid=0x0000000043c01800 nid=0x1b59 waiting on condition > > [0x0000000042359000..0x0000000042359a10] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab34bf7c0> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Ab > > stractQueuedSynchronizer.java:1925) > > at > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) > > at > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxMa > > nager.java:479) > > > > "Thread-7" daemon prio=10 tid=0x00002aaac80f7800 nid=0x1b58 runnable > > [0x000000004265c000..0x000000004265cd90] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.FileDispatcher.read0(Native Method) > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) > > at sun.nio.ch.IOUtil.read(IOUtil.java:206) > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) > > - locked <0x00002aaab34c4d98> (a java.lang.Object) > > at > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxMa > > nager.java:551) > > > > "Thread-6" daemon prio=10 tid=0x00002aaac80f5000 nid=0x1b57 waiting on > > condition [0x000000004255b000..0x000000004255bd10] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab34c44b0> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Ab > > stractQueuedSynchronizer.java:1925) > > at > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) > > at > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxMa > > nager.java:479) > > > > "Thread-5" daemon prio=10 tid=0x00002aaac80f3800 nid=0x1b56 runnable > > [0x000000004245a000..0x000000004245ac90] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.FileDispatcher.read0(Native Method) > > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) > > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233) > > at sun.nio.ch.IOUtil.read(IOUtil.java:206) > > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) > > - locked <0x00002aaab34bf440> (a java.lang.Object) > > at > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxMa > > nager.java:551) > > > > "Thread-2" daemon prio=10 tid=0x0000000043c00400 nid=0x1b53 waiting on > > condition [0x0000000042157000..0x0000000042157b10] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab34c5438> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Ab > > stractQueuedSynchronizer.java:1925) > > at > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) > > at > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxMa > > nager.java:479) > > > > "QuorumPeer:/0:0:0:0:0:0:0:0:2181" prio=10 tid=0x00002aaac80f2400 > nid=0x1b4f > > runnable [0x0000000042056000..0x0000000042056a90] > > java.lang.Thread.State: RUNNABLE > > at java.net.SocketInputStream.socketRead0(Native Method) > > at java.net.SocketInputStream.read(SocketInputStream.java:129) > > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > > at java.io.BufferedInputStream.read(BufferedInputStream.java:237) > > - locked <0x00002aaab34da038> (a java.io.BufferedInputStream) > > at java.io.DataInputStream.readInt(DataInputStream.java:370) > > at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:64) > > at > > > org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java: > > 65) > > at > > > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:109) > > at > org.apache.zookeeper.server.quorum.Follower.readPacket(Follower.java:109) > > - locked <0x00002aaab34d9fe8> (a org.apache.jute.BinaryInputArchive) > > at > > > org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:221) > > at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:450) > > > > "WorkerReceiver Thread" daemon prio=10 tid=0x00002aaac80f1800 nid=0x1b4e > > waiting on condition [0x0000000041f55000..0x0000000041f55a10] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab34be738> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Ab > > stractQueuedSynchronizer.java:1925) > > at > java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317) > > at > > > org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerReceiver > > .run(FastLeaderElection.java:186) > > at java.lang.Thread.run(Thread.java:619) > > > > "WorkerSender Thread" daemon prio=10 tid=0x00002aaac80f1000 nid=0x1b4d > > waiting on condition [0x0000000041ae5000..0x0000000041ae5d90] > > java.lang.Thread.State: WAITING (parking) > > at sun.misc.Unsafe.park(Native Method) > > - parking to wait for <0x00002aaab34be868> (a > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > > at > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Ab > > stractQueuedSynchronizer.java:1925) > > at > > > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358) > > at > > > org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.r > > un(FastLeaderElection.java:287) > > at java.lang.Thread.run(Thread.java:619) > > > > "Thread-1" prio=10 tid=0x00002aaac80ec800 nid=0x1b4a runnable > > [0x00000000419e4000..0x00000000419e4d10] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) > > at > > > sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145) > > - locked <0x00002aaab34bcf70> (a java.lang.Object) > > at > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxMana > > ger.java:413) > > > > "NIOServerCxn.Factory:2181" daemon prio=10 tid=0x00002aaac8066c00 > nid=0x1af6 > > runnable [0x0000000041333000..0x0000000041333c90] > > java.lang.Thread.State: RUNNABLE > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:184) > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) > > - locked <0x00002aaab3489ae8> (a sun.nio.ch.Util$1) > > - locked <0x00002aaab3489ad0> (a java.util.Collections$UnmodifiableSet) > > - locked <0x00002aaab34893f8> (a sun.nio.ch.EPollSelectorImpl) > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) > > at > > > org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:142) > > > > "RMI TCP Accept-0" daemon prio=10 tid=0x00002aaac805d800 nid=0x1af4 > runnable > > [0x0000000041131000..0x0000000041131b90] > > java.lang.Thread.State: RUNNABLE > > at java.net.PlainSocketImpl.socketAccept(Native Method) > > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) > > - locked <0x00002aaab3488238> (a java.net.SocksSocketImpl) > > at java.net.ServerSocket.implAccept(ServerSocket.java:453) > > at java.net.ServerSocket.accept(ServerSocket.java:421) > > at > > > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.j > > ava:369) > > at > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341) > > at java.lang.Thread.run(Thread.java:619) > > > > "Low Memory Detector" daemon prio=10 tid=0x000000004398dc00 nid=0x1af3 > > runnable [0x0000000000000000..0x0000000000000000] > > java.lang.Thread.State: RUNNABLE > > > > "CompilerThread1" daemon prio=10 tid=0x000000004398b400 nid=0x1af2 > waiting > > on condition [0x0000000000000000..0x0000000040f2ea80] > > java.lang.Thread.State: RUNNABLE > > > > "CompilerThread0" daemon prio=10 tid=0x0000000043989800 nid=0x1af1 > waiting > > on condition [0x0000000000000000..0x0000000040e2da10] > > java.lang.Thread.State: RUNNABLE > > > > "Signal Dispatcher" daemon prio=10 tid=0x0000000043988000 nid=0x1af0 > > runnable [0x0000000000000000..0x0000000040d2da20] > > java.lang.Thread.State: RUNNABLE > > > > "Finalizer" daemon prio=10 tid=0x0000000043964c00 nid=0x1aef in > > Object.wait() [0x0000000041e54000..0x0000000041e54d10] > > java.lang.Thread.State: WAITING (on object monitor) > > at java.lang.Object.wait(Native Method) > > - waiting on <0x00002aaab34b1a18> (a java.lang.ref.ReferenceQueue$Lock) > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) > > - locked <0x00002aaab34b1a18> (a java.lang.ref.ReferenceQueue$Lock) > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) > > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) > > > > "Reference Handler" daemon prio=10 tid=0x0000000043964000 nid=0x1aee in > > Object.wait() [0x0000000041d53000..0x0000000041d53c90] > > java.lang.Thread.State: WAITING (on object monitor) > > at java.lang.Object.wait(Native Method) > > - waiting on <0x00002aaab349abe0> (a java.lang.ref.Reference$Lock) > > at java.lang.Object.wait(Object.java:485) > > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > > - locked <0x00002aaab349abe0> (a java.lang.ref.Reference$Lock) > > > > "main" prio=10 tid=0x00000000438ae800 nid=0x1ae4 in Object.wait() > > [0x0000000040406000..0x0000000040406ed0] > > java.lang.Thread.State: WAITING (on object monitor) > > at java.lang.Object.wait(Native Method) > > - waiting on <0x00002aaab3488a38> (a > > org.apache.zookeeper.server.quorum.QuorumPeer) > > at java.lang.Thread.join(Thread.java:1143) > > - locked <0x00002aaab3488a38> (a > > org.apache.zookeeper.server.quorum.QuorumPeer) > > at java.lang.Thread.join(Thread.java:1196) > > at > > > org.apache.zookeeper.server.quorum.QuorumPeerMain.runPeer(QuorumPeerMain.java: > > 116) > > at > > > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:86) > > > > "VM Thread" prio=10 tid=0x000000004395f400 nid=0x1aed runnable > > > > "GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000438b9000 nid=0x1ae5 > > runnable > > > > "GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000438ba000 nid=0x1ae6 > > runnable > > > > "GC task thread#2 (ParallelGC)" prio=10 tid=0x00000000438bb400 nid=0x1ae7 > > runnable > > > > "GC task thread#3 (ParallelGC)" prio=10 tid=0x00000000438bc800 nid=0x1ae8 > > runnable > > > > "GC task thread#4 (ParallelGC)" prio=10 tid=0x00000000438bdc00 nid=0x1ae9 > > runnable > > > > "GC task thread#5 (ParallelGC)" prio=10 tid=0x00000000438bf000 nid=0x1aea > > runnable > > > > "GC task thread#6 (ParallelGC)" prio=10 tid=0x00000000438c0400 nid=0x1aeb > > runnable > > > > "GC task thread#7 (ParallelGC)" prio=10 tid=0x00000000438c1800 nid=0x1aec > > runnable > > > > "VM Periodic Task Thread" prio=10 tid=0x00002aaac8065800 nid=0x1af5 > waiting > > on condition > > > > JNI global references: 895 > > > > > > On Tue, Jun 2, 2009 at 1:22 PM, Benjamin Reed <br...@yahoo-inc.com> > wrote: > > > >> can you attach the jstack output? it seems to be missing from your > email. > >> > >> ben > >> > >> > >> Satish Bhatti wrote: > >> > >>> I am running a 5 node ZooKeeper cluster and I noticed that one of them > has > >>> very high CPU usage: > >>> > >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > >>> 6883 infact 22 0 725m 41m 4188 S 95 0.5 > >>> 5671:54 java > >>> > >>> It is not "doing anything" application-wise at this point, so I was > >>> wondering why the heck it's using up so much CPU! I have attached a > jstack > >>> logfile to this email. > >>> > >>> Satish > >>> > >>> > >> > >
