hi,
I have seen some supervisors daemon has more connections to Nimbus than
other supervisors.
*The special supervisor' jstack:*
==========
2015-09-15 15:22:16
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00007f722c002000 nid=0x34e5 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"process reaper" daemon prio=10 tid=0x00007f71fc028000 nid=0x5a2f runnable
[0x00007f72a4091000]
java.lang.Thread.State: RUNNABLE
at java.lang.UNIXProcess.waitForProcessExit(Native Method)
at java.lang.UNIXProcess.access$500(UNIXProcess.java:54)
at java.lang.UNIXProcess$4.run(UNIXProcess.java:227)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000f1f83888> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"DestroyJavaVM" prio=10 tid=0x00007f72a800b800 nid=0x6d59 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Thread-2" prio=10 tid=0x00007f72a8333000 nid=0x6de9 waiting on condition
[0x00007f72778f7000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f073c258> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at backtype.storm.event$event_manager$fn__2809.invoke(event.clj:39)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-1" prio=10 tid=0x00007f72a832e800 nid=0x6de8 waiting on condition
[0x00007f72779f8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f073e438> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at backtype.storm.event$event_manager$fn__2809.invoke(event.clj:39)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"CuratorFramework-0" daemon prio=10 tid=0x00007f72a8324000 nid=0x6de7
waiting on condition [0x00007f7277af9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f0708478> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:209)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:68)
at
org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:781)
at
org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:57)
at
org.apache.storm.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:275)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000f07082e8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"main-EventThread" daemon prio=10 tid=0x00007f72a8302000 nid=0x6de6 waiting
on condition [0x00007f7277efd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f07417e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.storm.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Locked ownable synchronizers:
- None
"main-SendThread(TS-BHTEST-03:2181)" daemon prio=10 tid=0x00007f72a8301800
nid=0x6de5 runnable [0x00007f7277cfb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000f073d6c0> (a sun.nio.ch.Util$2)
- locked <0x00000000f073d6b0> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00000000f073d290> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.apache.storm.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at
org.apache.storm.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Locked ownable synchronizers:
- None
"ConnectionStateManager-0" daemon prio=10 tid=0x00007f72a82f5800 nid=0x6de4
waiting on condition [0x00007f7277bfa000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f0707f08> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
at
org.apache.storm.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:208)
at
org.apache.storm.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:42)
at
org.apache.storm.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000000f0707d70> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"timer" daemon prio=10 tid=0x00007f72a8f42800 nid=0x6dde waiting on
condition [0x00007f7277ffe000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at backtype.storm.utils.Time.sleepUntil(Time.java:76)
at backtype.storm.utils.Time.sleep(Time.java:81)
at
backtype.storm.timer$mk_timer$fn__1521$fn__1522.invoke(timer.clj:57)
at backtype.storm.timer$mk_timer$fn__1521.invoke(timer.clj:42)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Service Thread" daemon prio=10 tid=0x00007f72a80b6000 nid=0x6d72 runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" daemon prio=10 tid=0x00007f72a80b3800 nid=0x6d71
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" daemon prio=10 tid=0x00007f72a80b1000 nid=0x6d70
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" daemon prio=10 tid=0x00007f72a80af000 nid=0x6d6f
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" daemon prio=10 tid=0x00007f72a808f800 nid=0x6d6e in
Object.wait() [0x00007f729d483000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000f00da668> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" daemon prio=10 tid=0x00007f72a808d800 nid=0x6d6d in
Object.wait() [0x00007f729d584000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000000f00e6d18> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"VM Thread" prio=10 tid=0x00007f72a8089800 nid=0x6d6c runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f72a8021000 nid=0x6d5a
runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f72a8023000 nid=0x6d5b
runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f72a8025000 nid=0x6d5c
runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f72a8027000 nid=0x6d5d
runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f72a8028800 nid=0x6d5e
runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f72a802a800 nid=0x6d5f
runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f72a802c800 nid=0x6d60
runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f72a802e000 nid=0x6d61
runnable
"GC task thread#8 (ParallelGC)" prio=10 tid=0x00007f72a8030000 nid=0x6d62
runnable
"GC task thread#9 (ParallelGC)" prio=10 tid=0x00007f72a8032000 nid=0x6d63
runnable
"GC task thread#10 (ParallelGC)" prio=10 tid=0x00007f72a8033800 nid=0x6d64
runnable
"GC task thread#11 (ParallelGC)" prio=10 tid=0x00007f72a8035800 nid=0x6d65
runnable
"GC task thread#12 (ParallelGC)" prio=10 tid=0x00007f72a8037800 nid=0x6d66
runnable
"GC task thread#13 (ParallelGC)" prio=10 tid=0x00007f72a8039800 nid=0x6d67
runnable
"GC task thread#14 (ParallelGC)" prio=10 tid=0x00007f72a803b000 nid=0x6d68
runnable
"GC task thread#15 (ParallelGC)" prio=10 tid=0x00007f72a803d000 nid=0x6d69
runnable
"GC task thread#16 (ParallelGC)" prio=10 tid=0x00007f72a803f000 nid=0x6d6a
runnable
"GC task thread#17 (ParallelGC)" prio=10 tid=0x00007f72a8040800 nid=0x6d6b
runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f72a80c0800 nid=0x6d73 waiting
on condition
JNI global references: 233
===============
There are much more connections from the supervisor to Nimbus's thrift port
(6627)
* storm # netstat -nap | grep NIMBUS_HOST:6627 |wc -l*
113
However, there is only one conneciton in some other supervisors, and
comparing nodes, the special supervisor hosts one topology worker
consisting of 5 executors.
I want to know whether the # of connection is related with the number of
worker processes in the node? Maybe it is something internals, and I
appreciate it very much for your answers.
Thanks,
--
Bing Jiang
