Hi, I had written a MetricsConsumer for CloudWatch some time ago. The code would need reworking a bit, I might release it if I find the time.
For StormCrawler <http://stormcrawler.net/> topologies, our Elasticsearch module <https://github.com/DigitalPebble/storm-crawler/tree/master/external/elasticsearch> has a MetricsConsumer which sends the metrics to an ES index; we use Kibana to display the graphs and monitor the crawls. Julien On 31 May 2016 at 14:19, Matthew Lowe <[email protected]> wrote: > Thought you all might be interested. > > I have now got capacity monitoring working with AWS Cloudwatch. These are > 2 bolts within the same topology: > > ​ > > > #! /bin/bash > > API_KEY='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' > > # get all the topology id's > TOPOLOGY_IDS=`curl -s > http://XXXXXXXXXXXXXXXXXXXXXXXX:8080/api/v1/topology/summary | json_pp | > grep '"id"' | sed 's/.* : "\(.*\)".*/\1/'` > for id in ${TOPOLOGY_IDS} ; do > TOPOLOGY=`curl -s > http://XXXXXXXXXXXXXXXXXXXXXXX:8080/api/v1/topology/${id}?sys=false` > <http://XXXXXXXXXXXXXXXXXXXXXXX:8080/api/v1/topology/$%7Bid%7D?sys=false> > > # check the capacity for each task > CAPACITIES=`echo ${TOPOLOGY} | json_pp | grep '"capacity"' | sed 's/.* : > "\(.*\)".*/\1/'` > CAPACITIES_ARRAY=() > for capactiy in ${CAPACITIES} ; do > CAPACITIES_ARRAY+=("${capactiy}") > done > > # get all of the bolt ids > BOLT_IDS=`echo ${TOPOLOGY} | json_pp | grep '"boltId"' | sed 's/.* : > "\(.*\)".*/\1/'` > BOLT_IDS_ARRAY=() > for boltId in ${BOLT_IDS} ; do > BOLT_IDS_ARRAY+=("${boltId}") > done > > # send a trigger if needed > # capacity and bolt id arrays will always be the same size > for(( x=0; x<${#CAPACITIES_ARRAY[@]}; x++ )); do > TIMESTAMP=`date +"%Y-%m-%dT%H:%M:%S.%N"` > /usr/local/bin/aws cloudwatch put-metric-data --metric-name > "${BOLT_IDS_ARRAY[$x]}-capacity" --namespace "${id}" --value > "${CAPACITIES_ARRAY[$x]}" --timestamp "${TIMESTAMP}" > done > > > done > > On Mon, May 30, 2016 at 5:37 PM, Radhwane Chebaane < > [email protected]> wrote: > >> >> Using graphite API was so helpful for us, but since it doesn't support >> *Tags* introduced since *InfluxDB 0.9*, we created a *new metric >> Consumer <https://github.com/mathieuboniface/storm-metrics-influxdb>* >> that supports the new InfluxDB API instead of Grahpite API. *Tags* are >> important when filtering dashboard metrics based on components, bolt or >> worker name. >> >> Regards, >> Radhwane >> >> >> On 30/05/2016 17:19, Stephen Powis wrote: >> >> +1 for graphite and grafana via Verisign's plugin. >> >> Using graphite a few years ago was a real game changer for us, and more >> recently grafana to help build out dashboards instead of copy/pasting >> graphite urls around. Here's two different dashboards we have relating to >> our storm topologies. We're able to correlate information from all parts >> of our app, hardware monitoring metrics (via zabbix) and of course storm. >> Additionally we use seyren on top of graphite for our alerting as well. >> >> bolt specific dashboard <http://i.imgur.com/ftKtci5.png> >> >> Dashboard correlating lots of related information from various sources >> <http://i.imgur.com/t7yJ8d5.jpg> >> >> >> >> On Mon, May 30, 2016 at 9:13 AM, Radhwane Chebaane < >> <[email protected]>[email protected]> wrote: >> >>> Hi Matthew, >>> >>> We actually use the InfluxData <https://influxdata.com/> Stack >>> (InfluxDb + Grafana). >>> We send our data directly to a time-series database, *InfluxDB* >>> <https://influxdata.com/time-series-platform/influxdb/>. Then, we >>> visualize metrics with a customizable dashboard, *Grafana* >>> <http://grafana.org/>. >>> This way, you can have real-time metrics on your Storm topology. You may >>> also add custom metrics for enhanced monitoring. >>> >>> To export Storm metrics to InfluxDB you can use this *MetricsConsumer *which >>> is compatible with the latest version of InfluxDB and Storm 1.0.0: >>> https://github.com/mathieuboniface/storm-metrics-influxdb >>> >>> Or you can use the old Verisign plug-in with Graphite protocol: >>> https://github.com/verisign/storm-graphite >>> >>> Best regards, >>> Radhwane CHEBAANE >>> >>> >>> On 30/05/2016 14:47, Matthew Lowe wrote: >>> >>> Hello all. >>> >>> What kind of monitoring solutions do you use with storm? >>> >>> For example I have a bash script that reads the Json data from the REST UI >>> and alerts if there are any bolts with high capacities. >>> >>> It's only small and hacky, but I am genuinely interested to how you all >>> monitor your topologies. >>> >>> Best Regards >>> Matthew Lowe >>> >>> >>> >> >> > -- *Open Source Solutions for Text Engineering* http://www.digitalpebble.com http://digitalpebble.blogspot.com/ #digitalpebble <http://twitter.com/digitalpebble>
