Repository: storm Updated Branches: refs/heads/master 4f5fdd928 -> 2ed793d9b
STORM-2908 Document Metrics V2 for 2.0.0 * commit squashed by Jungtaek Lim <[email protected]> This closes #2628 Project: http://git-wip-us.apache.org/repos/asf/storm/repo Commit: http://git-wip-us.apache.org/repos/asf/storm/commit/2c61eba9 Tree: http://git-wip-us.apache.org/repos/asf/storm/tree/2c61eba9 Diff: http://git-wip-us.apache.org/repos/asf/storm/diff/2c61eba9 Branch: refs/heads/master Commit: 2c61eba95caa2600e978972fc1749379cc57c5e1 Parents: 4f5fdd9 Author: Aaron Gresch <[email protected]> Authored: Tue Apr 10 08:26:41 2018 -0500 Committer: Jungtaek Lim <[email protected]> Committed: Sat Apr 14 02:15:36 2018 +0900 ---------------------------------------------------------------------- conf/storm.yaml.example | 26 +++++++- docs/index.md | 3 +- docs/metrics_v2.md | 149 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 176 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/storm/blob/2c61eba9/conf/storm.yaml.example ---------------------------------------------------------------------- diff --git a/conf/storm.yaml.example b/conf/storm.yaml.example index 6d75963..6a8d3c5 100644 --- a/conf/storm.yaml.example +++ b/conf/storm.yaml.example @@ -80,4 +80,28 @@ # - class: "org.apache.storm.metric.FileBasedEventLogger" # - class: "org.mycompany.MyEventLogger" # arguments: -# endpoint: "event-logger.mycompany.org" \ No newline at end of file +# endpoint: "event-logger.mycompany.org" + +# Metrics v2 configuration (optional) +#storm.metrics.reporters: +# # Graphite Reporter +# - class: "org.apache.storm.metrics2.reporters.GraphiteStormReporter" +# daemons: +# - "supervisor" +# - "nimbus" +# - "worker" +# report.period: 60 +# report.period.units: "SECONDS" +# graphite.host: "localhost" +# graphite.port: 2003 +# +# # Console Reporter +# - class: "org.apache.storm.metrics2.reporters.ConsoleStormReporter" +# daemons: +# - "worker" +# report.period: 10 +# report.period.units: "SECONDS" +# filter: +# class: "org.apache.storm.metrics2.filters.RegexFilter" +# expression: ".*my_component.*emitted.*" + http://git-wip-us.apache.org/repos/asf/storm/blob/2c61eba9/docs/index.md ---------------------------------------------------------------------- diff --git a/docs/index.md b/docs/index.md index af7c182..b1d881f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -77,7 +77,8 @@ But small change will not affect the user experience. We will notify the user wh * [Distributed RPC](Distributed-RPC.html) * [Transactional topologies](Transactional-topologies.html) * [Hooks](Hooks.html) -* [Metrics](Metrics.html) +* [Metrics (Deprecated)](Metrics.html) +* [Metrics V2](metrics_v2.html) * [State Checkpointing](State-checkpointing.html) * [Windowing](Windowing.html) * [Joining Streams](Joins.html) http://git-wip-us.apache.org/repos/asf/storm/blob/2c61eba9/docs/metrics_v2.md ---------------------------------------------------------------------- diff --git a/docs/metrics_v2.md b/docs/metrics_v2.md new file mode 100644 index 0000000..e90ec76 --- /dev/null +++ b/docs/metrics_v2.md @@ -0,0 +1,149 @@ +--- +title: Metrics Reporting API v2 +layout: documentation +documentation: true +--- +Apache Storm version 1.2 introduced a new metrics system for reporting +internal statistics (e.g. acked, failed, emitted, transferred, disruptor queue metrics, etc.) as well as a +new API for user defined metrics. + +The new metrics system is based on [Dropwizard Metrics](http://metrics.dropwizard.io). + + +## User Defined Metrics +To allow users to define custom metrics, the following methods have been added to the `TopologyContext` +class, an instance of which is passed to spout's `open()` method and bolt's `prepare()` method: + + public Timer registerTimer(String name) + + public Histogram registerHistogram(String name) + + public Meter registerMeter(String name) + + public Counter registerCounter(String name) + + public Gauge registerGauge(String name, Gauge gauge) + +API documentation: [Timer](http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Timer.html), +[Histogram](http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Histogram.html), +[Meter](http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Meter.html), +[Counter](http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Counter.html), +[Gauge](http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/Gauge.html) + +Each of these methods takes a `name` parameter that acts as an identifier. When metrics are +registered, Storm will add additional information such as hostname, port, topology ID, etc. to form a unique metric +identifier. For example, if we register a metric named `myCounter` as follows: + +```java + Counter myCounter = topologyContext.registerCounter("myCounter"); +``` +the resulting name sent to metrics reporters will expand to: + +``` + storm.topology.{topology ID}.{hostname}.{component ID}.{task ID}.{worker port}-myCounter +``` + +The additional information allows for the unique identification of metrics for component instances across the cluster. + +*Important Note:* In order to ensure metric names can be reliably parsed, any `.` characters in name components will +be replaced with an underscore (`_`) character. For example, the hostname `storm.example.com` will appear as +`storm_example_com` in the metric name. This character substitution *is not applied to the user-supplied `name` parameter. + +### Example: Tuple Counter Bolt +The following example is a simple bolt implementation that will report the running total up tuples received by a bolt: + +```java +public class TupleCountingBolt extends BaseRichBolt { + private Counter tupleCounter; + @Override + public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) { + this.tupleCounter = context.registerCounter("tupleCount"); + } + + @Override + public void execute(Tuple input) { + this.tupleCounter.inc(); + } +} +``` + +## Metric Reporter Configuration + + For metrics to be useful they must be *reported*, in other words sent somewhere where they can be consumed and analyzed. + That can be as simple as writing them to a log file, sending them to a time series database, or exposing them via JMX. + + As of Storm 1.2.0 the following metric reporters are supported + + * Console Reporter (`org.apache.storm.metrics2.reporters.ConsoleStormReporter`): + Reports metrics to `System.out`. + * CSV Reporter (`org.apache.storm.metrics2.reporters.CsvReporter`): + Reports metrics to a CSV file. + * Ganglia Reporter (`org.apache.storm.metrics2.reporters.GagliaStormReporter`): + Reports metrics to a [Ganglia](http://ganglia.info) server. + * Graphite Reporter (`org.apache.storm.metrics2.reporters.GraphiteStormReporter`): + Reports metrics to a [Graphite](https://graphiteapp.org) server. + * JMX Reporter (`org.apache.storm.metrics2.reporters.JmxStormReporter`): + Exposes metrics via JMX. + + + Metrics reporters are configured in the `storm.yaml` file. By default, Storm will collect metrics but not "report" or + send the collected metrics anywhere. To enable metrics reporting, add a `storm.metrics.reporters` section to `storm.yaml` + and configure one or more reporters. + + The following example configuration sets up two reporters: a Graphite Reporter and a Console Reporter: + + ```yaml +storm.metrics.reporters: + # Graphite Reporter + - class: "org.apache.storm.metrics2.reporters.GraphiteStormReporter" + daemons: + - "supervisor" + - "nimbus" + - "worker" + report.period: 60 + report.period.units: "SECONDS" + graphite.host: "localhost" + graphite.port: 2003 + + # Console Reporter + - class: "org.apache.storm.metrics2.reporters.ConsoleStormReporter" + daemons: + - "worker" + report.period: 10 + report.period.units: "SECONDS" + filter: + class: "org.apache.storm.metrics2.filters.RegexFilter" + expression: ".*my_component.*emitted.*" + +``` + +Each reporter section begins with a `class` parameter representing the fully-qualified class name of the reporter +implementation. The `daemons` section determines which daemons the reporter will apply to (in the example Graphite +Reporter is configured to report metrics from all Storm daemons, while the Console reporter will only report worker and +topology metrics). + +Many reporter implementations are *scheduled*, meaning they report metrics at regular intervals. The reporting interval +is determined by the `report.period` and `report.period.units` parameters. + +Reporters can also be configured with an optional filter that determines which metrics get reported. Storm includes the +`org.apache.storm.metrics2.filters.RegexFilter` filter which uses a regular expression to determine which metrics get +reported. Custom filters can be created by implementing the `org.apache.storm.metrics2.filters.StormMetricFilter` +interface: + +```java +public interface StormMetricsFilter extends MetricFilter { + + /** + * Called after the filter is instantiated. + * @param config A map of the properties from the 'filter' section of the reporter configuration. + */ + void prepare(Map<String, Object> config); + + /** + * Returns true if the given metric should be reported. + */ + boolean matches(String name, Metric metric); + +} +``` +
