Hi Manu,
Can you share your system configuration to see if I can get the stats using
your setup? I have copy and pasted the output from SOL to show you what I
see.
jack@node-1:/mnt/storm-benchmark$ bin/stormbench -storm
/mnt/storm/bin/storm -jar
./target/storm-benchmark-0.1.0-jar-with-dependencies.jar -conf
./conf/sol.yaml -c topology.workers=2 storm.benchmark.tools.Runner
storm.benchmark.benchmarks.SOL
Running: /mnt/jdk1.7.0_51/bin/java -client
-Dstorm.options=topology.workers%3D2 -Dstorm.home=/mnt/storm
-Dstorm.log.dir=/mnt/storm/logs
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
-Dstorm.conf.file=sol.yaml -cp
/mnt/storm/storm-wordcount-benchmark-0.11.0-SNAPSHOT-jar-with-dependencies.jar:/mnt/storm/lib/jline-2.11.jar:/mnt/storm/lib/log4j-over-slf4j-1.6.6.jar:/mnt/storm/lib/slf4j-api-1.7.5.jar:/mnt/storm/lib/logback-core-1.0.13.jar:/mnt/storm/lib/logback-classic-1.0.13.jar:/mnt/storm/lib/jgrapht-core-0.9.0.jar:/mnt/storm/lib/disruptor-2.10.1.jar:/mnt/storm/lib/commons-logging-1.1.3.jar:/mnt/storm/lib/snakeyaml-1.11.jar:/mnt/storm/lib/chill-java-0.3.5.jar:/mnt/storm/lib/objenesis-1.2.jar:/mnt/storm/lib/minlog-1.2.jar:/mnt/storm/lib/asm-4.0.jar:/mnt/storm/lib/reflectasm-1.07-shaded.jar:/mnt/storm/lib/kryo-2.21.jar:/mnt/storm/lib/carbonite-1.4.0.jar:/mnt/storm/lib/json-simple-1.1.jar:/mnt/storm/lib/commons-lang-2.5.jar:/mnt/storm/lib/commons-exec-1.1.jar:/mnt/storm/lib/tools.cli-0.2.4.jar:/mnt/storm/lib/math.numeric-tower-0.0.1.jar:/mnt/storm/lib/tools.logging-0.2.3.jar:/mnt/storm/lib/jetty-util-6.1.26.jar:/mnt/storm/lib/jetty-6.1.26.jar:/mnt/storm/lib/ring-servlet-0.3.11.jar:/mnt/storm/lib/ring-jetty-adapter-0.3.11.jar:/mnt/storm/lib/clj-stacktrace-0.2.2.jar:/mnt/storm/lib/ring-devel-0.3.11.jar:/mnt/storm/lib/hiccup-0.3.6.jar:/mnt/storm/lib/servlet-api-2.5.jar:/mnt/storm/lib/commons-fileupload-1.2.1.jar:/mnt/storm/lib/commons-io-2.4.jar:/mnt/storm/lib/commons-codec-1.6.jar:/mnt/storm/lib/ring-core-1.1.5.jar:/mnt/storm/lib/clout-1.0.1.jar:/mnt/storm/lib/tools.macro-0.1.0.jar:/mnt/storm/lib/core.incubator-0.1.0.jar:/mnt/storm/lib/compojure-1.1.3.jar:/mnt/storm/lib/joda-time-2.0.jar:/mnt/storm/lib/clj-time-0.4.1.jar:/mnt/storm/lib/clojure-1.5.1.jar:/mnt/storm/lib/storm-core-0.9.5.jar:./target/storm-benchmark-0.1.0-jar-with-dependencies.jar:/mnt/storm-benchmark/conf:/mnt/storm/bin
-Dlog4j.configurationFile=/mnt/storm/log4j2.xml
-Dstorm.jar=./target/storm-benchmark-0.1.0-jar-with-dependencies.jar
storm.benchmark.tools.Runner storm.benchmark.benchmarks.SOL
204 [main] INFO storm.benchmark.tools.Runner - running benchmark
storm.benchmark.benchmarks.SOL
460 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to
master yet. Submitting jar...
466 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar
./target/storm-benchmark-0.1.0-jar-with-dependencies.jar to assigned
location:
/tmp/storm/nimbus/inbox/stormjar-77bc1f1f-b621-4c3b-89d2-448caf7d70a7.jar
598 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded
topology jar to assigned location:
/tmp/storm/nimbus/inbox/stormjar-77bc1f1f-b621-4c3b-89d2-448caf7d70a7.jar
598 [main] INFO backtype.storm.StormSubmitter - Submitting topology SOL
in distributed mode with conf
{"drpc.worker.threads":64,"topology.state.synchronization.timeout.secs":60,"topology.executor.send.buffer.size":1024,"topology.worker.childopts":null,"storm.messaging.netty.max_retries":300,"topology.receiver.buffer.size":8,"topology.max.spout.pending":200,"storm.zookeeper.retry.intervalceiling.millis":30000,"metrics.enabled":true,"zmq.hwm":0,"topology.message.timeout.secs":30,"topology.max.task.parallelism":null,"topology.spout.wait.strategy":"backtype.storm.spout.SleepSpoutWaitStrategy","supervisor.slots.ports":[6700,6701,6702,6703],"topology.level":3,"storm.zookeeper.servers":["node-1"],"transactional.zookeeper.servers":null,"supervisor.childopts":"-Xmx256m","nimbus.host":"node-1","storm.cluster.mode":"distributed","ui.childopts":"-Xmx768m","component.spout_num":4,"topology.disruptor.wait.strategy":"com.lmax.disruptor.BlockingWaitStrategy","topology.enable.message.timeouts":true,"
logviewer.appender.name
":"A1","storm.local.mode.zmq":false,"drpc.request.timeout.secs":600,"topology.multilang.serializer":"backtype.storm.multilang.JsonSerializer","storm.zookeeper.root":"\/storm","storm.thrift.transport":"backtype.storm.security.auth.SimpleTransportPlugin","topology.debug":false,"supervisor.enable":true,"nimbus.thrift.port":6627,"
topology.sleep.spout.wait.strategy.time.ms
":1,"nimbus.reassign":true,"storm.zookeeper.connection.timeout":15000,"supervisor.heartbeat.frequency.secs":5,"topology.worker.shared.thread.pool.size":4,"zmq.threads":1,"topology.acker.executors":4,"
topology.name
":"SOL","dev.zookeeper.path":"\/tmp\/dev-storm-zookeeper","transactional.zookeeper.root":"\/transactional","worker.heartbeat.frequency.secs":1,"nimbus.supervisor.timeout.secs":60,"nimbus.thrift.max_buffer_size":1048576,"storm.local.dir":"storm-local","task.refresh.poll.secs":10,"supervisor.worker.timeout.secs":30,"topology.tick.tuple.freq.secs":null,"storm.messaging.netty.client_worker_threads":1,"nimbus.file.copy.expiration.secs":600,"nimbus.task.timeout.secs":30,"topology.builtin.metrics.bucket.size.secs":60,"storm.messaging.transport":"backtype.storm.messaging.netty.Context","topology.transfer.buffer.size":1024,"drpc.childopts":"-Xmx768m","topology.workers":2,"supervisor.monitor.frequency.secs":3,"transactional.zookeeper.port":null,"storm.messaging.netty.server_worker_threads":1,"topology.max.error.report.per.interval":5,"storm.messaging.netty.transfer.batch.size":262144,"metrics.time":300000,"storm.messaging.netty.buffer_size":5242880,"storm.messaging.netty.min_wait_ms":100,"nimbus.childopts":"-Xmx1024m","topology.skip.missing.kryo.registrations":false,"topology.trident.batch.emit.interval.millis":500,"message.size":100,"worker.childopts":"-Xmx768m","topology.stats.sample.rate":0.05,"topology.environment":null,"storm.zookeeper.port":2181,"supervisor.worker.start.timeout.secs":120,"nimbus.cleanup.inbox.freq.secs":600,"topology.kryo.factory":"backtype.storm.serialization.DefaultKryoFactory","task.heartbeat.frequency.secs":3,"topology.fall.back.on.java.serialization":true,"topology.worker.receiver.thread.count":1,"logviewer.port":8000,"ui.port":8080,"storm.zookeeper.retry.times":5,"topology.tuple.serializer":"backtype.storm.serialization.types.ListDelegateSerializer","storm.messaging.netty.max_wait_ms":1000,"topology.error.throttle.interval.secs":10,"nimbus.task.launch.secs":120,"drpc.port":3772,"storm.zookeeper.session.timeout":20000,"drpc.invocations.port":3773,"logviewer.childopts":"-Xmx128m","java.library.path":"\/usr\/local\/lib:\/opt\/local\/lib:\/usr\/lib","storm.meta.serialization.delegate":"backtype.storm.serialization.DefaultSerializationDelegate","metrics.path":"reports","
storm.messaging.netty.flush.check.interval.ms
":10,"zmq.linger.millis":5000,"nimbus.topology.validator":"backtype.storm.nimbus.DefaultTopologyValidator","nimbus.monitor.freq.secs":10,"metrics.poll":60000,"topology.executor.receive.buffer.size":1024,"topology.tasks":null,"storm.zookeeper.retry.interval":1000,"nimbus.inbox.jar.expiration.secs":3600,"topology.classpath":null,"drpc.queue.size":128,"component.bolt_num":8}
895 [main] INFO backtype.storm.StormSubmitter - Finished submitting
topology: SOL
911 [main] INFO storm.benchmark.metrics.MetricsCollectorConfig - writing
out storm config into .yaml file
915 [main] INFO storm.benchmark.metrics.BasicMetricsCollector - writing
out metrics headers
[time(s),total_slots,used_slots,workers,tasks,executors,transferred
(messages),throughput (messages/s),throughput (MB/s),spout_throughput
(MB/s),spout_executors,spout_transferred (messages),spout_acked
(messages),spout_throughput
(messages/s),spout_avg_complete_latency(ms),spout_max_complete_latency(ms)]
into .csv file
60979 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt1
60979 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt1
60979 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt1
60979 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt1
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt1
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt1
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt1
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt1
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt2
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt2
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt2
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt2
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt2
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt2
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt2
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: bolt2
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: spout
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: spout
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: spout
60980 [main] WARN storm.benchmark.metrics.BasicMetricsCollector - executor
stats not found for component: spout
60981 [main] INFO storm.benchmark.metrics.BasicMetricsCollector - writing
out metrics results [60,16,2,2,24,24,0,0,0.0,0.000,0,0,0,0,0.0,0.0] into
.csv file
121014 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
121014 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
121014 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
121014 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
121014 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: spout
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: spout
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: spout
121015 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: spout
121016 [main] INFO storm.benchmark.metrics.BasicMetricsCollector - writing
out metrics results [120,16,2,2,24,24,0,0,0.0,0.000,0,0,0,0,0.0,0.0] into
.csv file
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt1
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
181047 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: bolt2
181048 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: spout
181048 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: spout
181048 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: spout
181048 [main] WARN storm.benchmark.metrics.BasicMetricsCollector -
executor stats not found for component: spout
181048 [main] INFO storm.benchmark.metrics.BasicMetricsCollector - writing
out metrics results [180,16,2,2,24,24,0,0,0.0,0.000,0,0,0,0,0.0,0.0] into
.csv file
Thanks for your help.
Jack
On Tue, Jul 21, 2015 at 5:34 AM, Manu Zhang <[email protected]> wrote:
> Hi Jack,
>
> I've been successfully getting metrics data running storm-benchmark
> wordcount on storm 0.9.5.
> Maybe you can file a jira and describe how to reproduce your issue.
>
> Manu
>
> On Tue, Jul 21, 2015 at 9:33 AM, Jack <[email protected]> wrote:
>
> > Hi Manu,
> >
> > I am running version storm 0.9.5 with your wordcount benchmark with 4
> > workers, 4 spouts, 4 splits, 4 counts and 2 ack executor bolts. I set the
> > polling interval to 5 seconds and total running time to 3 minutes.
> >
> > Jack
> >
> > Jack
> >
> > On Mon, Jul 20, 2015 at 6:09 PM, Manu Zhang <[email protected]>
> > wrote:
> >
> > > Jack,
> > >
> > > Which Storm version are you using ? And what's the setup of your
> > > configurations and topologies ?
> > >
> > > Manu
> > >
> > > On Tue, Jul 21, 2015 at 6:45 AM, Jack <[email protected]>
> wrote:
> > >
> > > > Hi Manu,
> > > >
> > > > Yes, there are executors running and the benchmark runs successfully.
> > The
> > > > curious thing is I can see the executors stats for each spout/bolt
> if I
> > > > query the REST API directly, but prefer not to do this.
> > > >
> > > > Jack
> > > >
> > > > Jack
> > > >
> > > > On Mon, Jul 20, 2015 at 3:35 PM, Manu Zhang <[email protected]
> >
> > > > wrote:
> > > >
> > > > > Hi Jack,
> > > > >
> > > > > Could you check if there are any executors up ?
> > > > >
> > > > > Thanks,
> > > > > Manu Zhang
> > > > >
> > > > > On Tue, Jul 21, 2015 at 1:30 AM, Jack <[email protected]>
> > > wrote:
> > > > >
> > > > > > I'm trying to get ExecutorStats from ExecutorSummary but it
> always
> > > > > returns
> > > > > > null. Is there some sort of configuration I need to set before I
> > can
> > > > get
> > > > > > those values? I also tried using
> > > > > > https://github.com/intel-hadoop/storm-benchmark which also tries
> > to
> > > > > > retrieve executor stats but this benchmark also returns null for
> > > those
> > > > > > values. Thanks
> > > > > >
> > > > >
> > > >
> > >
> >
>
