Jungtaek,
Thanks for your help.
Please find enclose the jstack file:
- I remove by hand all name related to my project (spout, bolt...).
- The running jdk was openJDK (not an oracle JDK).
I also increase the buffer size (topology.executor.receive.buffer.size
and topology.executor.send.buffer.size) to 32768 and lauch my topology
on 2 workers. Then the dead lock occurs after 3 days. At this moment
24577 points have been send.
Please feel free to contact me if you need further information.
Denis
Le 29/03/2016 11:11, Jungtaek Lim a écrit :
Denis,
Could you share your jstack output which contains locked information?
And I'm not sure but you may hit STORM-1027
<https://issues.apache.org/jira/browse/STORM-1027> which is resolved
at 0.10.0 and 0.9.6.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX
<[email protected] <mailto:[email protected]>> 님이 작성:
Hi all,
I profile my code and I see that the thread
<BoldName>MetricConsumer is locked and never unlocked. It is why
my metrics are never published.
More over, the problem is related to buffer size:
* topology.executor.receive.buffer.size=16 and
topology.executor.send.buffer.size=16 implies a blocking after 4
minutes.
* with a size=512, I am able to run my test for more than 1 hour.
Thanks for your help.
Denis
Le 18/03/2016 22:38, Abhishek Agarwal a écrit :
Do you get this problem if you use only the LoggingMetricsConsumer?
Excuse typos
On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX"
<[email protected] <mailto:[email protected]>> wrote:
Dear all,
I have a question about metrics and how they are consumed:
I launch my topology. During 3 or 4 minutes, I receive my
metrics. Then no more logs.
It's look like the process that is consuming the metrics has
a dead lock or is blocked by a time out.
Here comes some information about the context:
- I am using storm 0.9.4.
- Some metrics are defined by defaut (eg emit-count.default)
some other are customed.
- I am sure that my topology is still running.
- According to Storm UI, the number of messages emiting by
my bolts is increasing. In other words, metrics are still
computing and send somewhere.
- I try two instances of LoggingMetricsConsumer: in file
(backtype.storm.metric.LoggingMetricsConsumer) and in
graphite (com.verisign.storm.metrics.GraphiteMetricsConsumer).
- I can reproduce it on 2 different machines.
- No error are logged (worker, metric, supervisor).
- I am using storm since 1 year and I never seen this problem.
Thanks for your help.
Denis
---
L'absence de virus dans ce courrier électronique a été
vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
------------------------------------------------------------------------
Avast logo
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
L'absence de virus dans ce courrier électronique a été vérifiée
par le logiciel antivirus Avast.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel
antivirus Avast.
https://www.avast.com/antivirus
2016-04-05 11:41:19
Full thread dump OpenJDK 64-Bit Server VM (25.77-b03 mixed mode):
"Timer-2" #85 prio=5 os_prio=0 tid=0x0000688e08098800 nid=0x719c in
Object.wait() [0x0000688db2a5c000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000688f4e4ff3c8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Attach Listener" #84 daemon prio=9 os_prio=0 tid=0x0000688e94001000 nid=0x6fe2
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"OkHttp ConnectionPool" #83 daemon prio=5 os_prio=0 tid=0x0000688d9002e000
nid=0x6f42 in Object.wait() [0x0000688db2cb8000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000688f4e5292b8> (a com.squareup.okhttp.ConnectionPool)
at java.lang.Object.wait(Object.java:460)
at
com.squareup.okhttp.ConnectionPool.performCleanup(ConnectionPool.java:305)
- locked <0x0000688f4e5292b8> (a com.squareup.okhttp.ConnectionPool)
at
com.squareup.okhttp.ConnectionPool.runCleanupUntilPoolIsEmpty(ConnectionPool.java:242)
at com.squareup.okhttp.ConnectionPool.access$000(ConnectionPool.java:54)
at com.squareup.okhttp.ConnectionPool$1.run(ConnectionPool.java:97)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x0000688f4e4ff7e8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"Okio Watchdog" #82 daemon prio=5 os_prio=0 tid=0x0000688d8407a000 nid=0x6f41
in Object.wait() [0x0000688db2dd3000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:460)
at okio.AsyncTimeout.awaitTimeout(AsyncTimeout.java:309)
- locked <0x0000688f4e54fef0> (a java.lang.Class for okio.AsyncTimeout)
at okio.AsyncTimeout.access$000(AsyncTimeout.java:40)
at okio.AsyncTimeout$Watchdog.run(AsyncTimeout.java:272)
Locked ownable synchronizers:
- None
"Timer-1" #81 prio=5 os_prio=0 tid=0x0000688e08029000 nid=0x6f3f in
Object.wait() [0x0000688db3541000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000688f4e544ca8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Timer-0" #80 prio=5 os_prio=0 tid=0x0000688e08021000 nid=0x6f3e in
Object.wait() [0x0000688db371e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000688f4e506b18> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Send Total latency" #78 prio=5 os_prio=0 tid=0x0000688dc0003000 nid=0x6f3d in
Object.wait() [0x0000688db3a38000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000688f4e506578> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"DestroyJavaVM" #77 prio=5 os_prio=0 tid=0x000068926000d000 nid=0x6e90 waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Thread-32-disruptor-worker-transfer-queue" #74 prio=5 os_prio=0
tid=0x00006892618b2000 nid=0x6f30 runnable [0x0000688db3b64000]
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.allocateMemory(Native Method)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:127)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at
org.apache.storm.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffers.java:167)
at
org.apache.storm.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffers.java:151)
at
backtype.storm.messaging.netty.MessageBatch.buffer(MessageBatch.java:84)
at
backtype.storm.messaging.netty.MessageEncoder.encode(MessageEncoder.java:32)
at
org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:66)
at
org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at
org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at
org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.apache.storm.netty.channel.Channels.write(Channels.java:704)
at org.apache.storm.netty.channel.Channels.write(Channels.java:671)
at
org.apache.storm.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
at backtype.storm.messaging.netty.Client.flushMessages(Client.java:323)
at backtype.storm.messaging.netty.Client.send(Client.java:261)
- locked <0x0000688f4b880290> (a java.lang.Object)
at backtype.storm.utils.TransferDrainer.send(TransferDrainer.java:54)
at
backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610$fn__4611.invoke(worker.clj:338)
at
backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610.invoke(worker.clj:336)
at
backtype.storm.disruptor$clojure_handler$reify__1445.onEvent(disruptor.clj:58)
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:132)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:106)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-30-disruptor-executor[1 1]-send-queue" #72 prio=5 os_prio=0
tid=0x00006892618ad800 nid=0x6f2e runnable [0x0000688e60159000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b59d110> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-29-__system" #71 prio=5 os_prio=0 tid=0x00006892618ab800 nid=0x6f2d
waiting on condition [0x0000688db3f06000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b5d9ab8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.daemon.executor$fn__3439$fn__3451$fn__3498.invoke(executor.clj:755)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-28-disruptor-executor[-1 -1]-send-queue" #70 prio=5 os_prio=0
tid=0x00006892618a9800 nid=0x6f2c waiting on condition [0x0000688e602b8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b605fa8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-27-__metricscom.verisign.storm.metrics.GraphiteMetricsConsumer" #69
prio=5 os_prio=0 tid=0x00006892618a7800 nid=0x6f2b waiting on condition
[0x0000688e6048f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b652488> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.daemon.executor$fn__3439$fn__3451$fn__3498.invoke(executor.clj:755)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-26-disruptor-executor[27 27]-send-queue" #68 prio=5 os_prio=0
tid=0x00006892618a6800 nid=0x6f2a waiting on condition [0x0000688e60686000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b6060d0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-24-disruptor-executor[25 25]-send-queue" #66 prio=5 os_prio=0
tid=0x00006892618a2800 nid=0x6f28 waiting on condition [0x0000688e609ee000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b66f618> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-22-disruptor-executor[23 23]-send-queue" #64 prio=5 os_prio=0
tid=0x000068926189f000 nid=0x6f26 waiting on condition [0x0000688e60d40000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b6b9680> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-20-disruptor-executor[21 21]-send-queue" #62 prio=5 os_prio=0
tid=0x000068926189c800 nid=0x6f24 waiting on condition [0x0000688e61120000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b6062b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-18-disruptor-executor[19 19]-send-queue" #60 prio=5 os_prio=0
tid=0x0000689261899800 nid=0x6f22 waiting on condition [0x0000688e613a0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b7b3218> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-16-disruptor-executor[17 17]-send-queue" #58 prio=5 os_prio=0
tid=0x0000689261893800 nid=0x6f20 waiting on condition [0x0000688e616df000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b782928> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-14-disruptor-executor[15 15]-send-queue" #56 prio=5 os_prio=0
tid=0x000068926188e000 nid=0x6f1e waiting on condition [0x0000688e61a16000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b745588> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-12-disruptor-executor[13 13]-send-queue" #54 prio=5 os_prio=0
tid=0x000068926188b800 nid=0x6f1c waiting on condition [0x0000688e61d07000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b6a7c00> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-10-disruptor-executor[11 11]-send-queue" #52 prio=5 os_prio=0
tid=0x0000689261880000 nid=0x6f1a waiting on condition [0x0000688e61fe9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b629240> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-8-disruptor-executor[9 9]-send-queue" #50 prio=5 os_prio=0
tid=0x0000689261878800 nid=0x6f18 waiting on condition [0x0000688e622ad000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b832ba8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-6-disruptor-executor[7 7]-send-queue" #48 prio=5 os_prio=0
tid=0x0000689261874000 nid=0x6f16 waiting on condition [0x0000688e62581000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b7c45a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-4-disruptor-executor[5 5]-send-queue" #46 prio=5 os_prio=0
tid=0x0000689261872000 nid=0x6f14 waiting on condition [0x0000688e62901000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b7df7a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-2-disruptor-executor[3 3]-send-queue" #44 prio=5 os_prio=0
tid=0x0000689261867000 nid=0x6f12 waiting on condition [0x0000688e62b52000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b6594b0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:83)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:101)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"pool-5-thread-1" #33 prio=5 os_prio=0 tid=0x0000688e48001000 nid=0x6f11
sleeping[0x0000688e62d17000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.storm.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
at
org.apache.storm.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
at
org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"client-schedule-timer-1" #35 prio=5 os_prio=0 tid=0x00006892617e8800
nid=0x6f10 sleeping[0x0000688e62e89000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.storm.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:483)
at
org.apache.storm.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:392)
at
org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Thread-1-worker-receiver-thread-0" #43 daemon prio=5 os_prio=0
tid=0x00006892617ae000 nid=0x6f0f waiting on condition [0x0000688e631fe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b6e6fe8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at backtype.storm.messaging.netty.Server.recv(Server.java:232)
at
backtype.storm.messaging.loader$mk_receive_thread$fn__1466$fn__1467.invoke(loader.clj:33)
at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Netty-server-localhost-6700-boss-1" #42 prio=5 os_prio=0
tid=0x0000689261783800 nid=0x6f0e runnable [0x0000688e633c6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000688f4b713f78> (a sun.nio.ch.Util$2)
- locked <0x0000688f4b713f68> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000688f4b713b28> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at
org.apache.storm.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at
org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.apache.storm.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at
org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x0000688f4b713a00> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"Netty-server-localhost-6700-worker-1" #41 prio=5 os_prio=0
tid=0x0000689261785800 nid=0x6f0d runnable [0x0000688e634e7000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000688f4b70f470> (a sun.nio.ch.Util$2)
- locked <0x0000688f4b70f460> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000688f4b6e78b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.storm.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at
org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x0000688f4b6e7760> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"user-timer" #40 daemon prio=10 os_prio=0 tid=0x0000689261772800 nid=0x6f0c
runnable [0x0000688e636d0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
at
com.lmax.disruptor.AbstractMultithreadedClaimStrategy.waitForFreeSlotAt(AbstractMultithreadedClaimStrategy.java:99)
at
com.lmax.disruptor.AbstractMultithreadedClaimStrategy.incrementAndGet(AbstractMultithreadedClaimStrategy.java:49)
at com.lmax.disruptor.Sequencer.next(Sequencer.java:127)
at
backtype.storm.utils.DisruptorQueue.publishDirect(DisruptorQueue.java:181)
at backtype.storm.utils.DisruptorQueue.publish(DisruptorQueue.java:174)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:66)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:68)
at
backtype.storm.daemon.executor$setup_metrics_BANG_$fn__3291.invoke(executor.clj:275)
at
backtype.storm.timer$schedule_recurring$this__1647.invoke(timer.clj:99)
at backtype.storm.timer$mk_timer$fn__1630$fn__1631.invoke(timer.clj:50)
at backtype.storm.timer$mk_timer$fn__1630.invoke(timer.clj:42)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"executor-heartbeat-timer" #39 daemon prio=10 os_prio=0 tid=0x0000689261770800
nid=0x6f0b waiting on condition [0x0000688e6388d000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at backtype.storm.utils.Time.sleepUntil(Time.java:76)
at backtype.storm.utils.Time.sleep(Time.java:81)
at backtype.storm.timer$mk_timer$fn__1630$fn__1631.invoke(timer.clj:57)
at backtype.storm.timer$mk_timer$fn__1630.invoke(timer.clj:42)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"refresh-active-timer" #38 daemon prio=10 os_prio=0 tid=0x000068926176f000
nid=0x6f0a waiting on condition [0x0000688f102c7000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at backtype.storm.utils.Time.sleepUntil(Time.java:76)
at backtype.storm.utils.Time.sleep(Time.java:81)
at backtype.storm.timer$mk_timer$fn__1630$fn__1631.invoke(timer.clj:57)
at backtype.storm.timer$mk_timer$fn__1630.invoke(timer.clj:42)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"refresh-connections-timer" #37 daemon prio=10 os_prio=0 tid=0x000068926176c000
nid=0x6f09 waiting on condition [0x0000688e63992000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at backtype.storm.utils.Time.sleepUntil(Time.java:76)
at backtype.storm.utils.Time.sleep(Time.java:81)
at backtype.storm.timer$mk_timer$fn__1630$fn__1631.invoke(timer.clj:57)
at backtype.storm.timer$mk_timer$fn__1630.invoke(timer.clj:42)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"heartbeat-timer" #36 daemon prio=10 os_prio=0 tid=0x000068926176d000
nid=0x6f08 sleeping[0x0000688f10446000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at backtype.storm.utils.Time.sleepUntil(Time.java:76)
at backtype.storm.utils.Time.sleep(Time.java:81)
at backtype.storm.timer$mk_timer$fn__1630$fn__1631.invoke(timer.clj:57)
at backtype.storm.timer$mk_timer$fn__1630.invoke(timer.clj:42)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"client-boss-1" #34 prio=5 os_prio=0 tid=0x0000689261743800 nid=0x6f07 runnable
[0x0000688f105bf000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000688f4b618068> (a sun.nio.ch.Util$2)
- locked <0x0000688f4b618058> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000688f4b617e10> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.storm.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at
org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.apache.storm.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
at
org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x0000688f4b617d30> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"client-worker-1" #32 prio=5 os_prio=0 tid=0x000068926172e800 nid=0x6f06
runnable [0x0000688e63f0d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000688f4b73a9c0> (a sun.nio.ch.Util$2)
- locked <0x0000688f4b73a9b0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000688f4b73a0e0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.storm.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
at
org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:415)
at
org.apache.storm.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
at
org.apache.storm.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.apache.storm.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.apache.storm.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.apache.storm.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x0000688f4b739ff8> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"CuratorFramework-0" #31 daemon prio=5 os_prio=0 tid=0x00006892616e8800
nid=0x6f03 waiting on condition [0x0000688f1073e000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b5df450> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
at
org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:781)
at
org.apache.storm.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:57)
at
org.apache.storm.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:275)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x0000688f4b5e0e38> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"main-EventThread" #30 daemon prio=5 os_prio=0 tid=0x00006892616e8000
nid=0x6f02 waiting on condition [0x0000688f112a1000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b5c6650> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.storm.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Locked ownable synchronizers:
- None
"main-SendThread(ZOOKEEPER_SERVER:2181)" #29 daemon prio=5 os_prio=0
tid=0x00006892616b9000 nid=0x6f01 runnable [0x0000688e63d78000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000688f4b5c1ef0> (a sun.nio.ch.Util$2)
- locked <0x0000688f4b5c1ee0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000688f4b5c1a38> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.storm.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at
org.apache.storm.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Locked ownable synchronizers:
- None
"ConnectionStateManager-0" #28 daemon prio=5 os_prio=0 tid=0x00006892616c2000
nid=0x6f00 waiting on condition [0x0000688f1018a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000688f4b5df5c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
org.apache.storm.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:208)
at
org.apache.storm.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:42)
at
org.apache.storm.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x0000688f4b5dfa00> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
"RMI TCP Accept-0" #23 daemon prio=5 os_prio=0 tid=0x0000689260365800
nid=0x6eb3 runnable [0x0000688f11a19000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI TCP Accept-3333" #22 daemon prio=5 os_prio=0 tid=0x000068926035a800
nid=0x6eb2 runnable [0x0000688f11b2e000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"RMI TCP Accept-0" #21 daemon prio=5 os_prio=0 tid=0x0000689260342800
nid=0x6eb1 runnable [0x0000688f11c87000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
at
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
"Service Thread" #19 daemon prio=9 os_prio=0 tid=0x000068926011c800 nid=0x6eb0
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread11" #18 daemon prio=9 os_prio=0 tid=0x0000689260111800
nid=0x6eaf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread10" #17 daemon prio=9 os_prio=0 tid=0x000068926010f000
nid=0x6eae waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread9" #16 daemon prio=9 os_prio=0 tid=0x000068926010d000
nid=0x6ead waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread8" #15 daemon prio=9 os_prio=0 tid=0x000068926010a800
nid=0x6eac waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread7" #14 daemon prio=9 os_prio=0 tid=0x0000689260108800
nid=0x6eab waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread6" #13 daemon prio=9 os_prio=0 tid=0x0000689260106800
nid=0x6eaa waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread5" #12 daemon prio=9 os_prio=0 tid=0x0000689260104000
nid=0x6ea9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread4" #11 daemon prio=9 os_prio=0 tid=0x00006892600fa000
nid=0x6ea8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread3" #10 daemon prio=9 os_prio=0 tid=0x00006892600f7800
nid=0x6ea7 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread2" #9 daemon prio=9 os_prio=0 tid=0x00006892600f5800
nid=0x6ea6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #8 daemon prio=9 os_prio=0 tid=0x00006892600f3800
nid=0x6ea5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #7 daemon prio=9 os_prio=0 tid=0x00006892600f0800
nid=0x6ea4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"JDWP Event Helper Thread" #6 daemon prio=10 os_prio=0 tid=0x00006892600ee800
nid=0x6ea3 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"JDWP Transport Listener: dt_socket" #5 daemon prio=10 os_prio=0
tid=0x00006892600eb000 nid=0x6ea2 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00006892600d3000
nid=0x6ea1 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00006892600aa000 nid=0x6ea0 in
Object.wait() [0x0000688f28404000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000688f4b030848> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00006892600a5800
nid=0x6e9f in Object.wait() [0x0000688f2850b000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x0000688f4b034b50> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x000068926009b800 nid=0x6e9e runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000689260022000 nid=0x6e91
runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000689260024000 nid=0x6e92
runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000689260025800 nid=0x6e93
runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000689260027800 nid=0x6e94
runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x0000689260029800 nid=0x6e95
runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x000068926002b000 nid=0x6e96
runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x000068926002d000 nid=0x6e97
runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x000068926002f000 nid=0x6e98
runnable
"GC task thread#8 (ParallelGC)" os_prio=0 tid=0x0000689260030800 nid=0x6e99
runnable
"GC task thread#9 (ParallelGC)" os_prio=0 tid=0x0000689260032800 nid=0x6e9a
runnable
"GC task thread#10 (ParallelGC)" os_prio=0 tid=0x0000689260034000 nid=0x6e9b
runnable
"GC task thread#11 (ParallelGC)" os_prio=0 tid=0x0000689260036000 nid=0x6e9c
runnable
"GC task thread#12 (ParallelGC)" os_prio=0 tid=0x0000689260038000 nid=0x6e9d
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x0000689260368000 nid=0x6eb4 waiting
on condition
JNI global references: 7152
