jnioche opened a new pull request, #9158:
URL: https://github.com/apache/storm/pull/9158
Storm ships Console, CSV, JMX and Graphite reporters for the V2 metrics, and
`storm-metrics-prometheus` only pushes the cluster metrics to a Pushgateway.
With JMX or Graphite, the task dimensions end up flattened in names like
`storm.worker.<topology>.<host>.<component>.<task>.<port>-<metric>`, which have
to be mapped back to labels with regular expressions.
This adds `PrometheusStormReporter` to `storm-metrics-prometheus`, a V2
`StormReporter` which serves the metrics of each worker over HTTP for
Prometheus to scrape. It is a port of apache/stormcrawler#2230.
### How it works
The reporter reads the metrics of each task from the
`MetricRegistryProvider`, along with the task's dimensions. These become labels
directly (`topology`, `host`, `port`, `component`, `task`, `stream`), so a
metric reported by several tasks ends up in a single family, without any name
parsing:
```
fetcher_counter_total{component="fetcher",host="worker1",port="6700",scope="status_200",task="3",topology="crawl"}
1234.0
status_count{component="status",host="worker1",key="FETCHED",port="6700",task="4",topology="crawl"}
56789.0
storm_ack_count_total{component="status",host="worker1",port="6700",source_component="spout",stream="default",task="4",topology="crawl"}
42.0
```
- Scoped metrics (`fetcher_counter.status_200`) become the family
`fetcher_counter` with a `scope` label. Only the first dot separates the scope,
so `fetcher_counter.robots.fetched` has `scope="robots.fetched"`.
- Gauges whose value is a map become a family named after the whole metric
(`status.count` -> `status_count`) with a `key` label per entry.
- Storm's own metrics (`__ack-count-spout:default`) become `storm_ack_count`
with `source_component` and `stream` labels. The `.m1_rate` gauges registered
next to Storm's counters are skipped, since Prometheus computes rates from the
counters.
- Counters and meters are exposed as counters, histograms and timers as
summaries plus a `_max` and a `_mean` gauge. Timers are converted to seconds.
- A gauge which throws when read is skipped rather than failing the whole
scrape.
### Configuration
```yaml
topology.metrics.reporters:
- class: "org.apache.storm.metrics.prometheus.PrometheusStormReporter"
port.offset: 1000
```
Each worker listens on its port plus `port.offset` (7700 for a worker on
6700), using the `worker.port` system property set by the supervisor, so
Prometheus scrapes one target per slot. In local mode, where the worker port is
unknown, the workers share a single endpoint on `port` (9400 by default). The
JVM metrics (memory, GC, threads, `process_start_time_seconds`) are exposed as
well unless `jvm.metrics` is false, and the usual `filter` applies. See the
module's README; `docs/metrics_v2.md` lists the new reporter.
The module gains dependencies on `prometheus-metrics-exporter-httpserver`
and `prometheus-metrics-instrumentation-jvm` (1.9.0, as for the existing
Prometheus dependencies), added to `DEPENDENCY-LICENSES`.
### Testing
- `StormMetricsCollectorTest`: unit tests for the naming rules.
- `PrometheusStormReporterTest`: starts a real `StormMetricRegistry`,
registers a metric of each type (including a map gauge and one of Storm's rate
counters), scrapes the HTTP endpoint and checks the samples; also covers the
port selection.
### Known limitations
- Dropwizard histograms do not track a sum, so the summaries have no `_sum`;
use the `_mean` gauge instead.
- The source component of Storm's per-stream metrics is found by assuming
their names have two words (`ack-count`, `process-latency`...), which holds for
all of Storm's current ones.
- Splitting on the first dot puts Storm's worker system metrics into
families like `memory{scope="heap.used"}`. The JVM metrics cover the same
ground.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
