Hi all,
I have a simple Storm distributed cluster running a simple topology that
runs slightly modified version of the ExclamationTopology sample.
When using netty, the CPU usage max outs (goes to 2100% on 16-core/thread
machine) and the throughput drastically drops (~3000 TPS). When switching
to zeromq with the same configs, the CPU usage is normal and the throughput
is around 100,000 - 170,000.
*Worker Node Config:*
Intel Xeon CPU E5-2470 0 @ 2.30GHz - 8 cores (16 threads)
RAM - 24 GB
JDK - 1.7.0_45
Netty configs (# of client worker threads, # of server worker threads,
buffer size) are at default values.
Found a lot of references to this EPollArrayWrapper JDK bug [1], which
seems to be fixed in JDK >= 1.7.0_11. But a thread dump when CPU was
hogging had the following stack traces (Full thread-dump trace attached).
"New I/O boss #6" prio=10 tid=0x00007f08c839c800 nid=0x7638 runnable
[0x00007f08cc8fa000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000f8410cd8> (a sun.nio.ch.Util$2)
- locked <0x00000000f8410cc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f8410bb0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"New I/O worker #5" prio=10 tid=0x00007f08c839c000 nid=0x7637 runnable
[0x00007f08cc9fb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000f839b7d8> (a sun.nio.ch.Util$2)
- locked <0x00000000f839b7c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f839b6b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
So wondering if it is the same issue or maybe there is some configuration
I've done incorrectly?
Any idea why this is happening? Any clues, pointers would be much
appreciated.. :-).
[1] https://github.com/netty/netty/issues/327
Thanks,
Lasantha
2014-05-08 04:16:13
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.45-b08 mixed mode):
"Attach Listener" daemon prio=10 tid=0x00007f08c0316000 nid=0x854 waiting on
condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Hashed wheel timer #2" prio=10 tid=0x0000000002872800 nid=0x7639 waiting on
condition [0x00007f08cc7f9000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
at
org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:744)
"New I/O boss #6" prio=10 tid=0x00007f08c839c800 nid=0x7638 runnable
[0x00007f08cc8fa000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000f8410cd8> (a sun.nio.ch.Util$2)
- locked <0x00000000f8410cc8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f8410bb0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"New I/O worker #5" prio=10 tid=0x00007f08c839c000 nid=0x7637 runnable
[0x00007f08cc9fb000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000f839b7d8> (a sun.nio.ch.Util$2)
- locked <0x00000000f839b7c8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000f839b6b0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"DestroyJavaVM" prio=10 tid=0x0000000002490800 nid=0x65c1 waiting on condition
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"New I/O server boss #4" prio=10 tid=0x00007f08c032a000 nid=0x65fb runnable
[0x00007f08ccafc000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d069b4d0> (a sun.nio.ch.Util$2)
- locked <0x00000000d069b4e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d069b488> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.select(NioServerBoss.java:163)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"New I/O worker #3" prio=10 tid=0x00007f08c0355000 nid=0x65fa runnable
[0x00007f08ccbfd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d06a6798> (a sun.nio.ch.Util$2)
- locked <0x00000000d06a67a8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d06a6750> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"Thread-11" prio=10 tid=0x00007f08c8465800 nid=0x65f9 runnable
[0x00007f08cccfe000]
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d06a5000> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:87)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:76)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:77)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1577.invoke(disruptor.clj:89)
at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
"Thread-10" daemon prio=10 tid=0x00007f08c8464800 nid=0x65f8 waiting on
condition [0x00007f08ccdff000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d068d1f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at backtype.storm.messaging.netty.Server.recv(Server.java:95)
at
backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1590$fn__1591.invoke(loader.clj:38)
at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
"Thread-9-__system" prio=10 tid=0x00007f08c8617000 nid=0x65f7 waiting on
condition [0x00007f08ccf00000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d06a9eb8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:87)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:76)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:77)
at
backtype.storm.daemon.executor$eval3918$fn__3919$fn__3931$fn__3978.invoke(executor.clj:745)
at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
"Thread-8" prio=10 tid=0x00007f08c8daa000 nid=0x65f6 waiting on condition
[0x00007f08cd001000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d069d998> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176)
at
com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:87)
at
com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:54)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:76)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:77)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1577.invoke(disruptor.clj:89)
at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
"Thread-7-StockData" prio=10 tid=0x00007f08c8992800 nid=0x65f5 runnable
[0x00007f08cd102000]
java.lang.Thread.State: RUNNABLE
at backtype.storm.stats.RollingWindow$reify__2651.get(stats.clj:30)
at backtype.storm.stats$cleanup_rolling_window.invoke(stats.clj:55)
at clojure.core$map$fn__4087.invoke(core.clj:2432)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
- locked <0x00000000f8464878> (a clojure.lang.LazySeq)
at clojure.lang.LazySeq.seq(LazySeq.java:60)
- locked <0x00000000f8464878> (a clojure.lang.LazySeq)
at clojure.lang.RT.seq(RT.java:473)
at clojure.core$seq.invoke(core.clj:133)
at
backtype.storm.stats$update_rolling_window_set$iter__2709__2713$fn__2714.invoke(stats.clj:74)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
- locked <0x00000000f846d8c0> (a clojure.lang.LazySeq)
at clojure.lang.LazySeq.seq(LazySeq.java:60)
- locked <0x00000000f846d8c0> (a clojure.lang.LazySeq)
at clojure.lang.RT.seq(RT.java:473)
at clojure.core$seq.invoke(core.clj:133)
at clojure.core$dorun.invoke(core.clj:2725)
at clojure.core$doall.invoke(core.clj:2741)
at backtype.storm.stats$update_rolling_window_set.doInvoke(stats.clj:74)
at clojure.lang.RestFn.invoke(RestFn.java:439)
at clojure.lang.Atom.swap(Atom.java:65)
at clojure.core$swap_BANG_.invoke(core.clj:2110)
at backtype.storm.stats$transferred_tuples_BANG_.invoke(stats.clj:206)
at backtype.storm.daemon.task$mk_tasks_fn$fn__3372.invoke(task.clj:167)
at
backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$send_spout_msg__3882.invoke(executor.clj:479)
at
backtype.storm.daemon.executor$eval3848$fn__3849$fn$reify__3891.emit(executor.clj:523)
at
backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:49)
at
backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:63)
at
backtype.storm.spout.SpoutOutputCollector.emit(SpoutOutputCollector.java:72)
at storm.starter.spout.StampTimeSpout.nextTuple(StampTimeSpout.java:32)
at
backtype.storm.daemon.executor$eval3848$fn__3849$fn__3864$fn__3893.invoke(executor.clj:562)
at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
"Thread-6" prio=10 tid=0x00007f08c8615800 nid=0x65f4 runnable
[0x00007f08cd203000]
java.lang.Thread.State: RUNNABLE
at sun.misc.Unsafe.unpark(Native Method)
at java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:152)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1263)
at
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:460)
at
com.lmax.disruptor.BlockingWaitStrategy.signalAllWhenBlocking(BlockingWaitStrategy.java:123)
at com.lmax.disruptor.Sequencer.publish(Sequencer.java:227)
at com.lmax.disruptor.Sequencer.publish(Sequencer.java:197)
at backtype.storm.utils.DisruptorQueue.publish(DisruptorQueue.java:139)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:66)
at backtype.storm.disruptor$publish.invoke(disruptor.clj:68)
at
backtype.storm.daemon.worker$mk_transfer_fn$fn__4217.invoke(worker.clj:124)
at
backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3746.invoke(executor.clj:255)
at
backtype.storm.disruptor$clojure_handler$reify__1560.onEvent(disruptor.clj:58)
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:104)
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:78)
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:77)
at
backtype.storm.disruptor$consume_loop_STAR_$fn__1577.invoke(disruptor.clj:89)
at backtype.storm.util$async_loop$fn__384.invoke(util.clj:433)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
"Hashed wheel timer #1" prio=10 tid=0x00000000027a7800 nid=0x65f3 waiting on
condition [0x00007f08cd304000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.jboss.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
at
org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:744)
"New I/O boss #2" prio=10 tid=0x00007f08c8727000 nid=0x65f2 runnable
[0x00007f08cd405000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d06b04f8> (a sun.nio.ch.Util$2)
- locked <0x00000000d06b0508> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d06a54f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"New I/O worker #1" prio=10 tid=0x00007f08c89e6800 nid=0x65f1 runnable
[0x00007f08ce058000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d06a8838> (a sun.nio.ch.Util$2)
- locked <0x00000000d06a8848> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d06a87f0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at
org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"Thread-4" daemon prio=10 tid=0x00007f08c8c1c000 nid=0x65f0 waiting on
condition [0x00007f08cd506000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at backtype.storm.utils.Time.sleepUntil(Time.java:70)
at backtype.storm.utils.Time.sleep(Time.java:75)
at backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke(timer.clj:53)
at backtype.storm.timer$mk_timer$fn__1790.invoke(timer.clj:41)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
"Thread-3" daemon prio=10 tid=0x00007f08c8afc000 nid=0x65ef runnable
[0x00007f08cd606000]
java.lang.Thread.State: RUNNABLE
at java.lang.Object.getClass(Native Method)
at clojure.lang.Util.classOf(Util.java:95)
at
clojure.core.protocols$fn__5828$G__5823__5841.invoke(protocols.clj:13)
at clojure.core$reduce.invoke(core.clj:6030)
at clojure.core$into.invoke(core.clj:6077)
at backtype.storm.stats$value_rolling_window_set.invoke(stats.clj:88)
at
backtype.storm.stats$value_stats$iter__2877__2881$fn__2882$fn__2883.invoke(stats.clj:258)
at
backtype.storm.stats$value_stats$iter__2877__2881$fn__2882.invoke(stats.clj:257)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
- locked <0x00000000f8464a50> (a clojure.lang.LazySeq)
at clojure.lang.LazySeq.seq(LazySeq.java:60)
- locked <0x00000000f8464a50> (a clojure.lang.LazySeq)
at clojure.lang.RT.seq(RT.java:473)
at clojure.core$seq.invoke(core.clj:133)
at clojure.core$dorun.invoke(core.clj:2725)
at clojure.core$doall.invoke(core.clj:2741)
at backtype.storm.stats$value_stats.invoke(stats.clj:257)
at backtype.storm.stats$value_common_stats.invoke(stats.clj:263)
at backtype.storm.stats$value_spout_stats_BANG_.invoke(stats.clj:274)
at backtype.storm.stats$eval2899$fn__2900.invoke(stats.clj:282)
at clojure.lang.MultiFn.invoke(MultiFn.java:163)
at
backtype.storm.daemon.executor$mk_executor$reify__3789.render_stats(executor.clj:341)
at
backtype.storm.daemon.worker$do_executor_heartbeats$fn__4199.invoke(worker.clj:45)
at clojure.core$map$fn__4087.invoke(core.clj:2434)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
- locked <0x00000000f8464788> (a clojure.lang.LazySeq)
at clojure.lang.LazySeq.seq(LazySeq.java:60)
- locked <0x00000000f8464788> (a clojure.lang.LazySeq)
at clojure.lang.RT.seq(RT.java:473)
at clojure.core$seq.invoke(core.clj:133)
at clojure.core$apply.invoke(core.clj:601)
at
backtype.storm.daemon.worker$do_executor_heartbeats.doInvoke(worker.clj:45)
at clojure.lang.RestFn.invoke(RestFn.java:439)
at
backtype.storm.daemon.worker$eval4413$exec_fn__1102__auto____4414$fn__4417.invoke(worker.clj:367)
at
backtype.storm.timer$schedule_recurring$this__1807.invoke(timer.clj:92)
at backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke(timer.clj:48)
at backtype.storm.timer$mk_timer$fn__1790.invoke(timer.clj:41)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
"Thread-2" daemon prio=10 tid=0x000000000273e800 nid=0x65ee waiting on
condition [0x00007f08cd708000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at backtype.storm.utils.Time.sleepUntil(Time.java:70)
at backtype.storm.utils.Time.sleep(Time.java:75)
at backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke(timer.clj:53)
at backtype.storm.timer$mk_timer$fn__1790.invoke(timer.clj:41)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
"Thread-1" daemon prio=10 tid=0x000000000304f000 nid=0x65ed waiting on
condition [0x00007f08cd809000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at backtype.storm.utils.Time.sleepUntil(Time.java:70)
at backtype.storm.utils.Time.sleep(Time.java:75)
at backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke(timer.clj:53)
at backtype.storm.timer$mk_timer$fn__1790.invoke(timer.clj:41)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
"Thread-0" daemon prio=10 tid=0x000000000304e000 nid=0x65ec runnable
[0x00007f08cd90a000]
java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.list(Native Method)
at java.io.File.list(File.java:1116)
at java.io.File.listFiles(File.java:1201)
at backtype.storm.utils.VersionedStore.listDir(VersionedStore.java:178)
at
backtype.storm.utils.VersionedStore.getAllVersions(VersionedStore.java:136)
at
backtype.storm.utils.VersionedStore.mostRecentVersion(VersionedStore.java:60)
at
backtype.storm.utils.VersionedStore.mostRecentVersionPath(VersionedStore.java:48)
at backtype.storm.utils.LocalState.snapshot(LocalState.java:42)
- locked <0x00000000f84625c0> (a backtype.storm.utils.LocalState)
at backtype.storm.utils.LocalState.put(LocalState.java:64)
- locked <0x00000000f84625c0> (a backtype.storm.utils.LocalState)
at backtype.storm.daemon.worker$do_heartbeat.invoke(worker.clj:66)
at
backtype.storm.daemon.worker$eval4413$exec_fn__1102__auto____4414$heartbeat_fn__4415.invoke(worker.clj:354)
at
backtype.storm.timer$schedule_recurring$this__1807.invoke(timer.clj:92)
at backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke(timer.clj:48)
at backtype.storm.timer$mk_timer$fn__1790.invoke(timer.clj:41)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:744)
"CuratorFramework-1" prio=10 tid=0x00007f08c030f800 nid=0x65eb waiting on
condition [0x00007f08cda0b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d06aa088> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
com.netflix.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:537)
at
com.netflix.curator.framework.imps.CuratorFrameworkImpl.access$200(CuratorFrameworkImpl.java:50)
at
com.netflix.curator.framework.imps.CuratorFrameworkImpl$2.call(CuratorFrameworkImpl.java:177)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"ConnectionStateManager-0" prio=10 tid=0x00007f08c0325800 nid=0x65ea waiting on
condition [0x00007f08cdc0d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d06d2678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
at
com.netflix.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:145)
at
com.netflix.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:39)
at
com.netflix.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:90)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
"main-EventThread" daemon prio=10 tid=0x00007f08c0871800 nid=0x65e9 waiting on
condition [0x00007f08cdf57000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000d06b06a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
"main-SendThread(sr11:2181)" daemon prio=10 tid=0x00007f08c0307800 nid=0x65e8
runnable [0x00007f08cdb0c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000d06d6190> (a sun.nio.ch.Util$2)
- locked <0x00000000d06d61a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000d06b3e18> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
"Service Thread" daemon prio=10 tid=0x00007f08c0001000 nid=0x65df runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x0000000002544000 nid=0x65de waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x0000000002541000 nid=0x65dd waiting
on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x0000000002536800 nid=0x65dc runnable
[0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x000000000251f800 nid=0x65db in Object.wait()
[0x00007f08ceefe000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000d0021ea0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000000d0021ea0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
"Reference Handler" daemon prio=10 tid=0x000000000251b800 nid=0x65da in
Object.wait() [0x00007f08cefff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000d0021ed0> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000000d0021ed0> (a java.lang.ref.Reference$Lock)
"VM Thread" prio=10 tid=0x0000000002519000 nid=0x65d9 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x00000000024a6000 nid=0x65c2
runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x00000000024a8000 nid=0x65c3
runnable
"GC task thread#2 (ParallelGC)" prio=10 tid=0x00000000024aa000 nid=0x65c4
runnable
"GC task thread#3 (ParallelGC)" prio=10 tid=0x00000000024ab800 nid=0x65c5
runnable
"GC task thread#4 (ParallelGC)" prio=10 tid=0x00000000024ad800 nid=0x65c6
runnable
"GC task thread#5 (ParallelGC)" prio=10 tid=0x00000000024af800 nid=0x65c7
runnable
"GC task thread#6 (ParallelGC)" prio=10 tid=0x00000000024b1800 nid=0x65c8
runnable
"GC task thread#7 (ParallelGC)" prio=10 tid=0x00000000024b3000 nid=0x65c9
runnable
"GC task thread#8 (ParallelGC)" prio=10 tid=0x00000000024b5000 nid=0x65ca
runnable
"GC task thread#9 (ParallelGC)" prio=10 tid=0x00000000024b7000 nid=0x65cb
runnable
"GC task thread#10 (ParallelGC)" prio=10 tid=0x00000000024b8800 nid=0x65cc
runnable
"GC task thread#11 (ParallelGC)" prio=10 tid=0x00000000024ba800 nid=0x65cd
runnable
"GC task thread#12 (ParallelGC)" prio=10 tid=0x00000000024bc800 nid=0x65ce
runnable
"GC task thread#13 (ParallelGC)" prio=10 tid=0x00000000024be800 nid=0x65cf
runnable
"GC task thread#14 (ParallelGC)" prio=10 tid=0x00000000024c0000 nid=0x65d0
runnable
"GC task thread#15 (ParallelGC)" prio=10 tid=0x00000000024c2000 nid=0x65d1
runnable
"GC task thread#16 (ParallelGC)" prio=10 tid=0x00000000024c4000 nid=0x65d2
runnable
"GC task thread#17 (ParallelGC)" prio=10 tid=0x00000000024c5800 nid=0x65d3
runnable
"GC task thread#18 (ParallelGC)" prio=10 tid=0x00000000024c7800 nid=0x65d4
runnable
"GC task thread#19 (ParallelGC)" prio=10 tid=0x00000000024c9800 nid=0x65d5
runnable
"GC task thread#20 (ParallelGC)" prio=10 tid=0x00000000024cb000 nid=0x65d6
runnable
"GC task thread#21 (ParallelGC)" prio=10 tid=0x00000000024cd000 nid=0x65d7
runnable
"GC task thread#22 (ParallelGC)" prio=10 tid=0x00000000024cf000 nid=0x65d8
runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f08c000c000 nid=0x65e0 waiting on
condition
JNI global references: 511
