That is what i was suspecting, how do i get gc logs? right now in my logs i dont see any gc related logging.
On Thu, Oct 26, 2017 at 2:08 PM, Abraham Fine <af...@apache.org> wrote: > Another possibility... have you ruled out garbage collection related > pauses? > > Abe > > On Thu, Oct 26, 2017, at 13:57, rammohan ganapavarapu wrote: > > I was running ls command with in zkCli console from the same box. > > > > zkCli.sh --server localhost ls /node1, so no network involved. > > > > On Oct 26, 2017 12:37 PM, "Abraham Fine" <af...@apache.org> wrote: > > > > > Thanks for the jstack. I am not seeing anything unusual there. > > > > > > In your first message you mentioned that the command you executed is a > > > "local zk 'ls' command". By "local" do you mean that the command is > > > being executed from the same machine where the instance of zookeeper > you > > > are querying is running? It would be great to determine if the latency > > > is network related or more tied to zookeeper internals. > > > > > > Abe > > > > > > On Thu, Oct 26, 2017, at 10:57, rammohan ganapavarapu wrote: > > > > Abe, > > > > > > > > It is just 20kb file, not sure why its not able to deliver. I am just > > > > copying the jstack here. > > > > > > > > > > > > ### > > > > > 2017-10-24 04:09:49 > > > > > Full thread dump OpenJDK 64-Bit Server VM (25.101-b13 mixed mode): > > > > > "Attach Listener" #758568 daemon prio=9 os_prio=0 > > > tid=0x00007efbfc001000 > > > > > nid=0x10cfb waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "SyncThread:210" #74 prio=5 os_prio=0 tid=0x00007efba0036000 > > > nid=0x1a951 > > > > > waiting on condition [0x00007efbbaeef000] > > > > > java.lang.Thread.State: WAITING (parking) > > > > > at sun.misc.Unsafe.park(Native Method) > > > > > - parking to wait for <0x00000006c55eced0> (a > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject) > > > > > at java.util.concurrent.locks.LockSupport.park(LockSupport. > java:175) > > > > > at > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > > > > > at > > > > > java.util.concurrent.LinkedBlockingQueue.take( > > > LinkedBlockingQueue.java:442) > > > > > at > > > > > org.apache.zookeeper.server.SyncRequestProcessor.run( > > > SyncRequestProcessor.java:97) > > > > > "ObserverRequestProcessor:210" #73 prio=5 os_prio=0 > > > tid=0x00007efba0035800 > > > > > nid=0x1a950 waiting on condition [0x00007efbbaff0000] > > > > > java.lang.Thread.State: WAITING (parking) > > > > > at sun.misc.Unsafe.park(Native Method) > > > > > - parking to wait for <0x00000006c55ec2e8> (a > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject) > > > > > at java.util.concurrent.locks.LockSupport.park(LockSupport. > java:175) > > > > > at > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > ConditionObject.await(AbstractQueuedSynchronizer.java:2039) > > > > > at > > > > > java.util.concurrent.LinkedBlockingQueue.take( > > > LinkedBlockingQueue.java:442) > > > > > at > > > > > org.apache.zookeeper.server.quorum.ObserverRequestProcessor.run( > > > ObserverRequestProcessor.java:66) > > > > > "CommitProcessor:210" #72 prio=5 os_prio=0 tid=0x00007efba004d800 > > > > > nid=0x1a94f in Object.wait() [0x00007efbbb0f1000] > > > > > java.lang.Thread.State: WAITING (on object monitor) > > > > > at java.lang.Object.wait(Native Method) > > > > > at java.lang.Object.wait(Object.java:502) > > > > > at > > > > > org.apache.zookeeper.server.quorum.CommitProcessor.run( > > > CommitProcessor.java:80) > > > > > - locked <0x00000006c55eb8e0> (a > > > > > org.apache.zookeeper.server.quorum.CommitProcessor) > > > > > "RecvWorker:204" #40 daemon prio=5 os_prio=0 tid=0x00007efba8013000 > > > > > nid=0x1a415 runnable [0x00007efbbb1f2000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > at java.net.SocketInputStream.socketRead0(Native Method) > > > > > at java.net.SocketInputStream.socketRead(SocketInputStream. > java:116) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:170) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:141) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:223) > > > > > at java.io.DataInputStream.readInt(DataInputStream.java:387) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$ > RecvWorker.run( > > > QuorumCnxManager.java:747) > > > > > "SendWorker:204" #39 daemon prio=5 os_prio=0 tid=0x00007efba8011000 > > > > > nid=0x1a414 waiting on condition [0x00007efbbb2f3000] > > > > > java.lang.Thread.State: TIMED_WAITING (parking) > > > > > at sun.misc.Unsafe.park(Native Method) > > > > > - parking to wait for <0x00000005c018c938> (a > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject) > > > > > at java.util.concurrent.locks.LockSupport.parkNanos( > > > LockSupport.java:215) > > > > > at > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > > at > > > > > java.util.concurrent.ArrayBlockingQueue.poll( > > > ArrayBlockingQueue.java:418) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue( > > > QuorumCnxManager.java:831) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > access$500(QuorumCnxManager.java:62) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$ > SendWorker.run( > > > QuorumCnxManager.java:667) > > > > > "RecvWorker:203" #38 daemon prio=5 os_prio=0 tid=0x00007efba800f800 > > > > > nid=0x1a413 runnable [0x00007efbbb3f4000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > at java.net.SocketInputStream.socketRead0(Native Method) > > > > > at java.net.SocketInputStream.socketRead(SocketInputStream. > java:116) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:170) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:141) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:223) > > > > > at java.io.DataInputStream.readInt(DataInputStream.java:387) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$ > RecvWorker.run( > > > QuorumCnxManager.java:747) > > > > > "SendWorker:203" #37 daemon prio=5 os_prio=0 tid=0x00007efba800d800 > > > > > nid=0x1a412 waiting on condition [0x00007efbbb4f5000] > > > > > java.lang.Thread.State: TIMED_WAITING (parking) > > > > > at sun.misc.Unsafe.park(Native Method) > > > > > - parking to wait for <0x00000005c018bf00> (a > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject) > > > > > at java.util.concurrent.locks.LockSupport.parkNanos( > > > LockSupport.java:215) > > > > > at > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > > at > > > > > java.util.concurrent.ArrayBlockingQueue.poll( > > > ArrayBlockingQueue.java:418) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue( > > > QuorumCnxManager.java:831) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > access$500(QuorumCnxManager.java:62) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$ > SendWorker.run( > > > QuorumCnxManager.java:667) > > > > > "RecvWorker:202" #36 daemon prio=5 os_prio=0 tid=0x00007efba800c800 > > > > > nid=0x1a411 runnable [0x00007efbbb6f7000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > at java.net.SocketInputStream.socketRead0(Native Method) > > > > > at java.net.SocketInputStream.socketRead(SocketInputStream. > java:116) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:170) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:141) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:223) > > > > > at java.io.DataInputStream.readInt(DataInputStream.java:387) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$ > RecvWorker.run( > > > QuorumCnxManager.java:747) > > > > > "SendWorker:202" #35 daemon prio=5 os_prio=0 tid=0x00007efba800b800 > > > > > nid=0x1a410 waiting on condition [0x00007efbbb5f6000] > > > > > java.lang.Thread.State: TIMED_WAITING (parking) > > > > > at sun.misc.Unsafe.park(Native Method) > > > > > - parking to wait for <0x00000005c018b4c8> (a > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject) > > > > > at java.util.concurrent.locks.LockSupport.parkNanos( > > > LockSupport.java:215) > > > > > at > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > > at > > > > > java.util.concurrent.ArrayBlockingQueue.poll( > > > ArrayBlockingQueue.java:418) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue( > > > QuorumCnxManager.java:831) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > access$500(QuorumCnxManager.java:62) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$ > SendWorker.run( > > > QuorumCnxManager.java:667) > > > > > "RecvWorker:201" #32 daemon prio=5 os_prio=0 tid=0x00007efba8009800 > > > > > nid=0x1a40d runnable [0x00007efbbb7f8000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > at java.net.SocketInputStream.socketRead0(Native Method) > > > > > at java.net.SocketInputStream.socketRead(SocketInputStream. > java:116) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:170) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:141) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:223) > > > > > at java.io.DataInputStream.readInt(DataInputStream.java:387) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$ > RecvWorker.run( > > > QuorumCnxManager.java:747) > > > > > "SendWorker:201" #31 daemon prio=5 os_prio=0 tid=0x00007efba8008800 > > > > > nid=0x1a40c waiting on condition [0x00007efbbb8f9000] > > > > > java.lang.Thread.State: TIMED_WAITING (parking) > > > > > at sun.misc.Unsafe.park(Native Method) > > > > > - parking to wait for <0x00000005c016f830> (a > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject) > > > > > at java.util.concurrent.locks.LockSupport.parkNanos( > > > LockSupport.java:215) > > > > > at > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > > at > > > > > java.util.concurrent.ArrayBlockingQueue.poll( > > > ArrayBlockingQueue.java:418) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue( > > > QuorumCnxManager.java:831) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > access$500(QuorumCnxManager.java:62) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$ > SendWorker.run( > > > QuorumCnxManager.java:667) > > > > > "RecvWorker:200" #30 daemon prio=5 os_prio=0 tid=0x00007efba8008000 > > > > > nid=0x1a3e5 runnable [0x00007efbbbafb000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > at java.net.SocketInputStream.socketRead0(Native Method) > > > > > at java.net.SocketInputStream.socketRead(SocketInputStream. > java:116) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:170) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:141) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:223) > > > > > at java.io.DataInputStream.readInt(DataInputStream.java:387) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$ > RecvWorker.run( > > > QuorumCnxManager.java:747) > > > > > "SendWorker:200" #29 daemon prio=5 os_prio=0 tid=0x00007efba8007000 > > > > > nid=0x1a3e4 waiting on condition [0x00007efbbb9fa000] > > > > > java.lang.Thread.State: TIMED_WAITING (parking) > > > > > at sun.misc.Unsafe.park(Native Method) > > > > > - parking to wait for <0x00000005c016f910> (a > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject) > > > > > at java.util.concurrent.locks.LockSupport.parkNanos( > > > LockSupport.java:215) > > > > > at > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > > at > > > > > java.util.concurrent.ArrayBlockingQueue.poll( > > > ArrayBlockingQueue.java:418) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue( > > > QuorumCnxManager.java:831) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > access$500(QuorumCnxManager.java:62) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$ > SendWorker.run( > > > QuorumCnxManager.java:667) > > > > > "QuorumPeer[myid=210]/0:0:0:0:0:0:0:0:2181" #22 prio=5 os_prio=0 > > > > > tid=0x00007efc6c2f7000 nid=0x1a3df runnable [0x00007efbbbbfc000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > at java.net.SocketInputStream.socketRead0(Native Method) > > > > > at java.net.SocketInputStream.socketRead(SocketInputStream. > java:116) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:170) > > > > > at java.net.SocketInputStream.read(SocketInputStream.java:141) > > > > > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > > > > > at java.io.BufferedInputStream.read(BufferedInputStream.java:265) > > > > > - locked <0x00000005c01981b0> (a java.io.BufferedInputStream) > > > > > at java.io.DataInputStream.readInt(DataInputStream.java:387) > > > > > at org.apache.jute.BinaryInputArchive.readInt( > > > BinaryInputArchive.java:63) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumPacket. > > > deserialize(QuorumPacket.java:83) > > > > > at > > > > > org.apache.jute.BinaryInputArchive.readRecord( > > > BinaryInputArchive.java:108) > > > > > at org.apache.zookeeper.server.quorum.Learner.readPacket( > > > Learner.java:152) > > > > > - locked <0x00000005c019a228> (a org.apache.jute. > BinaryInputArchive) > > > > > at > > > > > org.apache.zookeeper.server.quorum.Observer.observeLeader( > > > Observer.java:75) > > > > > at org.apache.zookeeper.server.quorum.QuorumPeer.run( > > > QuorumPeer.java:727) > > > > > "WorkerReceiver[myid=210]" #25 daemon prio=5 os_prio=0 > > > > > tid=0x00007efc6c2fc800 nid=0x1a3de waiting on condition > > > [0x00007efbbbcfd000] > > > > > java.lang.Thread.State: TIMED_WAITING (parking) > > > > > at sun.misc.Unsafe.park(Native Method) > > > > > - parking to wait for <0x00000005c016fbf0> (a > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject) > > > > > at java.util.concurrent.locks.LockSupport.parkNanos( > > > LockSupport.java:215) > > > > > at > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > > at > > > > > java.util.concurrent.ArrayBlockingQueue.poll( > > > ArrayBlockingQueue.java:418) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager.pollRecvQueue( > > > QuorumCnxManager.java:883) > > > > > at > > > > > org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > Messenger$WorkerReceiver.run(FastLeaderElection.java:205) > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > "WorkerSender[myid=210]" #24 daemon prio=5 os_prio=0 > > > > > tid=0x00007efc6c2f9000 nid=0x1a3dd waiting on condition > > > [0x00007efbbbdfe000] > > > > > java.lang.Thread.State: TIMED_WAITING (parking) > > > > > at sun.misc.Unsafe.park(Native Method) > > > > > - parking to wait for <0x00000005c016fec0> (a > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > ConditionObject) > > > > > at java.util.concurrent.locks.LockSupport.parkNanos( > > > LockSupport.java:215) > > > > > at > > > > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) > > > > > at > > > > > java.util.concurrent.LinkedBlockingQueue.poll( > > > LinkedBlockingQueue.java:467) > > > > > at > > > > > org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > Messenger$WorkerSender.run(FastLeaderElection.java:362) > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > "192.168.72.250:3888" #23 prio=5 os_prio=0 tid=0x00007efc6c2c2800 > > > > > nid=0x1a3dc runnable [0x00007efc001d3000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > at java.net.PlainSocketImpl.socketAccept(Native Method) > > > > > at > > > > > java.net.AbstractPlainSocketImpl.accept( > AbstractPlainSocketImpl.java: > > > 409) > > > > > at java.net.ServerSocket.implAccept(ServerSocket.java:545) > > > > > at java.net.ServerSocket.accept(ServerSocket.java:513) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumCnxManager$ > > > Listener.run(QuorumCnxManager.java:491) > > > > > "NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181" #21 daemon prio=5 > > > os_prio=0 > > > > > tid=0x00007efc6c305000 nid=0x1a3db runnable [0x00007efc002d4000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) > > > > > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) > > > > > at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl. > java:93) > > > > > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > > > > > - locked <0x00000005c0008620> (a sun.nio.ch.Util$2) > > > > > - locked <0x00000005c0008610> (a java.util.Collections$ > > > UnmodifiableSet) > > > > > - locked <0x00000005c00084d8> (a sun.nio.ch.EPollSelectorImpl) > > > > > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > > > > > at > > > > > org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > NIOServerCnxnFactory.java:178) > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > "PurgeTask" #20 daemon prio=5 os_prio=0 tid=0x00007efc6c2a1800 > > > nid=0x1a39a > > > > > in Object.wait() [0x00007efc003d5000] > > > > > java.lang.Thread.State: TIMED_WAITING (on object monitor) > > > > > at java.lang.Object.wait(Native Method) > > > > > at java.util.TimerThread.mainLoop(Timer.java:552) > > > > > - locked <0x00000005c0002f30> (a java.util.TaskQueue) > > > > > at java.util.TimerThread.run(Timer.java:505) > > > > > "RMI TCP Accept-0" #19 daemon prio=5 os_prio=0 > tid=0x00007efc6c23d800 > > > > > nid=0x1a398 runnable [0x00007efc009f7000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > at java.net.PlainSocketImpl.socketAccept(Native Method) > > > > > at > > > > > java.net.AbstractPlainSocketImpl.accept( > AbstractPlainSocketImpl.java: > > > 409) > > > > > at java.net.ServerSocket.implAccept(ServerSocket.java:545) > > > > > at java.net.ServerSocket.accept(ServerSocket.java:513) > > > > > at > > > > > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop( > > > TCPTransport.java:400) > > > > > at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run( > > > TCPTransport.java:372) > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > "Service Thread" #17 daemon prio=9 os_prio=0 tid=0x00007efc6c106800 > > > > > nid=0x1a397 runnable [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C1 CompilerThread11" #16 daemon prio=9 os_prio=0 > > > tid=0x00007efc6c0fc000 > > > > > nid=0x1a396 waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C1 CompilerThread10" #15 daemon prio=9 os_prio=0 > > > tid=0x00007efc6c0fa000 > > > > > nid=0x1a395 waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C1 CompilerThread9" #14 daemon prio=9 os_prio=0 > tid=0x00007efc6c0f8000 > > > > > nid=0x1a394 waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C1 CompilerThread8" #13 daemon prio=9 os_prio=0 > tid=0x00007efc6c0f5800 > > > > > nid=0x1a393 waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C2 CompilerThread7" #12 daemon prio=9 os_prio=0 > tid=0x00007efc6c0f3800 > > > > > nid=0x1a392 waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C2 CompilerThread6" #11 daemon prio=9 os_prio=0 > tid=0x00007efc6c0f1800 > > > > > nid=0x1a391 waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C2 CompilerThread5" #10 daemon prio=9 os_prio=0 > tid=0x00007efc6c0ef800 > > > > > nid=0x1a390 waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C2 CompilerThread4" #9 daemon prio=9 os_prio=0 > tid=0x00007efc6c0e5800 > > > > > nid=0x1a38f waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C2 CompilerThread3" #8 daemon prio=9 os_prio=0 > tid=0x00007efc6c0e3000 > > > > > nid=0x1a38e waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 > tid=0x00007efc6c0e1000 > > > > > nid=0x1a38d waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 > tid=0x00007efc6c0df000 > > > > > nid=0x1a38c waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 > tid=0x00007efc6c0d1800 > > > > > nid=0x1a38b waiting on condition [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "Signal Dispatcher" #4 daemon prio=9 os_prio=0 > tid=0x00007efc6c0cf800 > > > > > nid=0x1a38a runnable [0x0000000000000000] > > > > > java.lang.Thread.State: RUNNABLE > > > > > "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007efc6c0a5800 > > > nid=0x1a389 > > > > > in Object.wait() [0x00007efc28c06000] > > > > > java.lang.Thread.State: WAITING (on object monitor) > > > > > at java.lang.Object.wait(Native Method) > > > > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) > > > > > - locked <0x00000005c01402e0> (a java.lang.ref.ReferenceQueue$ > Lock) > > > > > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) > > > > > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) > > > > > "Reference Handler" #2 daemon prio=10 os_prio=0 > tid=0x00007efc6c0a1000 > > > > > nid=0x1a388 in Object.wait() [0x00007efc28d07000] > > > > > java.lang.Thread.State: WAITING (on object monitor) > > > > > at java.lang.Object.wait(Native Method) > > > > > at java.lang.Object.wait(Object.java:502) > > > > > at java.lang.ref.Reference.tryHandlePending(Reference.java:191) > > > > > - locked <0x00000005c0007c38> (a java.lang.ref.Reference$Lock) > > > > > at java.lang.ref.Reference$ReferenceHandler.run( > Reference.java:153) > > > > > "main" #1 prio=5 os_prio=0 tid=0x00007efc6c008800 nid=0x1a379 in > > > > > Object.wait() [0x00007efc75e17000] > > > > > java.lang.Thread.State: WAITING (on object monitor) > > > > > at java.lang.Object.wait(Native Method) > > > > > - waiting on <0x00000005c0007d28> (a > > > > > org.apache.zookeeper.server.quorum.QuorumPeer) > > > > > at java.lang.Thread.join(Thread.java:1245) > > > > > - locked <0x00000005c0007d28> (a > > > > > org.apache.zookeeper.server.quorum.QuorumPeer) > > > > > at java.lang.Thread.join(Thread.java:1319) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain. > > > runFromConfig(QuorumPeerMain.java:152) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain. > initializeAndRun( > > > QuorumPeerMain.java:111) > > > > > at > > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain.main( > > > QuorumPeerMain.java:78) > > > > > "VM Thread" os_prio=0 tid=0x00007efc6c097800 nid=0x1a387 runnable > > > > > "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007efc6c01e000 > > > > > nid=0x1a37a runnable > > > > > "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007efc6c01f800 > > > > > nid=0x1a37b runnable > > > > > "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007efc6c021800 > > > > > nid=0x1a37c runnable > > > > > "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007efc6c023000 > > > > > nid=0x1a37d runnable > > > > > "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007efc6c025000 > > > > > nid=0x1a37e runnable > > > > > "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007efc6c027000 > > > > > nid=0x1a37f runnable > > > > > "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007efc6c028800 > > > > > nid=0x1a380 runnable > > > > > "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007efc6c02a800 > > > > > nid=0x1a381 runnable > > > > > "GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007efc6c02c800 > > > > > nid=0x1a382 runnable > > > > > "GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007efc6c02e000 > > > > > nid=0x1a383 runnable > > > > > "GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007efc6c030000 > > > > > nid=0x1a384 runnable > > > > > "GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007efc6c031800 > > > > > nid=0x1a385 runnable > > > > > "GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007efc6c033800 > > > > > nid=0x1a386 runnable > > > > > "VM Periodic Task Thread" os_prio=0 tid=0x00007efc6c254800 > nid=0x1a399 > > > > > waiting on condition > > > > > JNI global references: 238 > > > > > > > > > > > > ### > > > > > > > > On Thu, Oct 26, 2017 at 9:13 AM, Abraham Fine <af...@apache.org> > wrote: > > > > > > > > > Maybe it is a limitation of the mailing list infrastructure, but > I'm > > > not > > > > > getting them. > > > > > Would it be possible to put the jstack output in a gist and link > > > > > to it here? > > > > > Abe > > > > > > > > > > > > > > > On Thu, Oct 26, 2017, at 09:00, rammohan ganapavarapu wrote: > > > > > > Hmm, it is attached in my sent mail but any way i am attaching > > > again.> > > > > > > On Thu, Oct 26, 2017 at 8:58 AM, Abraham Fine > > > > > > <af...@apache.org> wrote:>> Hi Ram- > > > > > >> > > > > > >> I don't see any jstack output attached to the email. > > > > > >> > > > > > >> Thanks, > > > > > >> Abe > > > > > >> > > > > > >> > > > > > >> > > > > > >> On Wed, Oct 25, 2017, at 18:17, rammohan ganapavarapu wrote: > > > > > >> > Abe, > > > > > >> > > > > > > >> > Under /node1 i have 30 child nodes but each child node > doesn't > > > > > >> > have>> > any more nodes. I did took jstack at the time of > > > latency, > > > > > please > > > > > >> > find>> > the attached. It is happening with most of the > servers > > > i > > > > > didn't > > > > > >> > check>> > all of them.> > > > > > >> > Thanks, > > > > > >> > Ram > > > > > >> > > > > > > >> > On Wed, Oct 25, 2017 at 4:06 PM, Abraham Fine > > > > > >> > <af...@apache.org> wrote:>> That's strange. ZooKeeper's > reads > > > > > >> > happen locally usually quickly.>> >> > > > > > >> >> Can you tell us anything else about the issue? Could you > > > > > >> >> possibly>> >> provide a jstack output from a server that > is > > > slow > > > > > to respond? > > > > > >> >> Does>> this happen with all of the servers? Does it > matter if > > > > > >> >> the node>> >> you are>> reading from has many children? > > > > > >> >> > > > > > >> >> Thanks, > > > > > >> >> Abe > > > > > >> >> > > > > > >> >> > > > > > >> >> On Wed, Oct 25, 2017, at 09:18, rammohan ganapavarapu wrote: > > > > > >> >> > Hi, > > > > > >> >> > > > > > > >> >> > We have a multi region zk cluster, we are observing > latency > > > > > >> >> > for>> >> > just>> > local > > > > > >> >> > zk 'ls' command intermittently nothing in logs dont know > how > > > > > >> >> > to>> >> > find>> > reason for latency, any one have > any > > > > > thoughts? > > > > > >> >> > > > > > > >> >> > zkCli.sh ls /node1 > > > > > >> >> > > > > > > >> >> > real 0m13.089s --When latency > > > > > >> >> > real 0m0.689s -- No latency > > > > > >> >> > > > > > > >> >> > Thanks, > > > > > >> >> > Ram > > > > > >> > > > > > > > > > > > > > >
