Hi Raphael,
I tested it with wordcounttopology under examples.
conf.registerMetricsConsumer(LoggingMetricsConsumer.class,
2);
I do see the metrics added to the logs/metrics.log. metrics.log
should be present by default under storm/logs dir.
-Harsha
On Mon, Sep 22, 2014, at 09:24 AM, Raphael Hsieh wrote:
Thanks Harsha and Otis for your prompt responses.
I'm looking to somehow log these metrics to use for an in-house
monitoring system. I don't want to get user provided metrics
just yet.
>From what I've gathered from the big data cookbook is that I
just want to create a metrics consumer to read these metrics
and print it out to a log file. In order to do this I have
added to my config:
config.registerMetricsConsumer(LoggingMetricsConsumer.class,
2);
which should create a loggingMetricsConsumer with a parallelism
of 2 (I believe). I was lead to believe that these logs would
be put in a file called "metrics.log". However after adding
this to my topology I have been unable to find such a log.
If someone could explain to me what I might be missing that
would be great.
Thanks!
On Fri, Sep 19, 2014 at 4:27 PM, Gezim Musliaj
<[1]gmusl...@gmail.com> wrote:
Hey Otis, I was just registered at sematext and I can say that
this is what I have been looking for.I have just one question,
what about the delays between the SPM and the Storm Cluster (if
they do exist), whats the worst case? I mean because these
metrics are not calculated locally, but using an internet
connection.
Thanks !
On Sat, Sep 20, 2014 at 1:15 AM, Otis Gospodnetic
<[2]otis.gospodne...@gmail.com> wrote:
Raphael,
Not sure if this is what you are after, but [3]SPM will collect
and graph all Storm metrics, let you do alerting and anomaly
detection on them, etc. If you want to graph custom metrics
(e.g. something from your bolts), you can send them in as
[4]custom metrics and again graph them, alert on them, do
anomaly detection on them, stick them on dashboards, etc. If
you want to emit events from your bolts, you can [5]send events
to SPM, too, or you can send them to [6]Logsene... can be handy
for correlation with alerts and performance graphs when
troubleshooting. Here are some Storm metrics graph:
[7]http://blog.sematext.com/2014/01/30/announcement-apache-stor
m-monitoring-in-spm/
I hope this helps.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log
Management
Solr & Elasticsearch Support * [8]http://sematext.com/
On Fri, Sep 19, 2014 at 6:12 PM, Raphael Hsieh
<[9]raffihs...@gmail.com> wrote:
Hi,
Using Storm/Trident, how do I register a metrics consumer to
log the data I get in the StormUI ?
I want to look at historical data of my topology, for example
the execute latency of the topology over time, as this would
give me good insight as to where things might be going wrong
when the system breaks.
I have been following the steps outlined in the BigData
CookBook
here: [10]http://www.bigdata-cookbook.com/post/72320512609/stor
m-metrics-how-to
However I am not wanting to create my own metrics, instead I
just want to log the metrics that already exist built in to
Storm. It is unclear to me how I am supposed to go about doing
that.
Thanks
--
Raphael Hsieh
--
Raphael Hsieh
References
1. mailto:gmusl...@gmail.com
2. mailto:otis.gospodne...@gmail.com
3. http://sematext.com/spm/
4. https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics
5. https://sematext.atlassian.net/wiki/display/PUBSPM/Events+Integration
6. http://www.sematext.com/logsene/
7.
http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
8. http://sematext.com/
9. mailto:raffihs...@gmail.com
10. http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
