jnioche opened a new pull request, #2230:
URL: https://github.com/apache/stormcrawler/pull/2230
StormCrawler can register its metrics with the Storm V2 API
(`stormcrawler.metrics.version: "v2"` or `"both"`), but Storm only ships
Console, CSV, JMX and Graphite reporters for them. Storm's
`storm-metrics-prometheus` module only pushes the Nimbus cluster summary
metrics to a Pushgateway. With JMX or Graphite, the dimensions end up flattened
in names like
`storm.worker.<topology>.<host>.<component>.<task>.<port>-fetcher_counter.status_200`,
which have to be mapped back to labels with regular expressions.
This adds a new module, `external/prometheus`, with `PrometheusReporter`, a
V2 `StormReporter` which serves the metrics of each worker over HTTP for
Prometheus to scrape.
### How it works
Storm passes its reporters a `MetricRegistryProvider`, whose
`getTaskMetrics()` gives the metrics of each task under their short name along
with the task's dimensions. These become labels directly (`topology`, `host`,
`port`, `component`, `task`, `stream`), so a metric reported by several tasks
ends up in a single family, without any name parsing:
```
fetcher_counter_total{component="fetcher",host="worker1",port="6700",scope="status_200",task="3",topology="crawl"}
1234.0
status_count{component="status",host="worker1",key="FETCHED",port="6700",task="4",topology="crawl"}
56789.0
storm_ack_count_total{component="status",host="worker1",port="6700",source_component="spout",stream="default",task="4",topology="crawl"}
42.0
```
- Scoped metrics (`fetcher_counter.status_200`) become the family
`fetcher_counter` with a `scope` label. Only the first dot separates the scope,
so `fetcher_counter.robots.fetched` has `scope="robots.fetched"`.
- Gauges whose value is a map, such as `status.count` from
`StatusMetricsBolt`, become a family named after the whole metric
(`status_count`) with a `key` label per entry.
- Storm's own metrics (`__ack-count-spout:default`) become `storm_ack_count`
with `source_component` and `stream` labels. The `.m1_rate` gauges Storm
registers next to its counters are skipped, since Prometheus computes rates
from the counters.
- Counters and meters are exposed as counters, histograms and timers as
summaries plus a `_max` and a `_mean` gauge. Timers are converted to seconds.
- A gauge which throws when read is skipped rather than failing the whole
scrape.
### Configuration
```yaml
stormcrawler.metrics.version: "v2"
topology.metrics.reporters:
- class: "org.apache.stormcrawler.prometheus.PrometheusReporter"
port.offset: 1000
```
Each worker listens on its port plus `port.offset` (7700 for a worker on
6700), using the `worker.port` system property set by the supervisor, so
Prometheus scrapes one target per slot. In local mode, where the worker port is
unknown, the workers share a single endpoint on `port` (9400 by default). The
JVM metrics (memory, GC, threads, `process_start_time_seconds`) are exposed as
well unless `jvm.metrics` is false, and the usual Storm `filter` applies. See
the module's README.
### Testing
- Unit tests for the naming rules and the port selection.
- `PrometheusReporterTest` starts a real `StormMetricRegistry`, registers a
metric of each type (including a map gauge and one of Storm's rate counters),
scrapes the HTTP endpoint and checks the samples.
- Outside of the tests, I fed two scrapes of metrics named as in a typical
crawl topology into VictoriaMetrics and checked that the PromQL queries of a
set of Grafana dashboards (fetching, spout, status updates, parsing, Storm
capacity, latency, JVMs) all returned the expected series.
It has not run against a live cluster yet.
### Known limitations
- Dropwizard histograms do not track a sum, so the summaries have no `_sum`;
use the `_mean` gauge instead.
- The source component of Storm's per-stream metrics is found by assuming
their names have two words (`ack-count`, `process-latency`...), which holds for
all of Storm's current ones.
- Splitting on the first dot suits StormCrawler's metric names, but puts
Storm's own system metrics into families like `memory{scope="heap.used"}`. The
JVM metrics cover the same ground.
- `cache.hits` and `cache.misses` from `AbstractStatusUpdaterBolt` are reset
on read, which is fixed separately in #2229.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
