janhoy opened a new pull request, #5244: URL: https://github.com/apache/solr/pull/5244
https://issues.apache.org/jira/browse/SOLR-18536 Solr can push metrics over OTLP, but nothing documented how to get them into Prometheus under the names the shipped dashboard and alert rules query. This adds that as a third artifact in `solr/monitoring/`, and lets the dev stack exercise either route. ## Changes - **`solr/monitoring/otel-collector-config.yaml`** (new, operator-facing example — not a production config): OTLP in on 4317/4318, Prometheus format out on `:9464`, raw metrics and traces forwarded to an Aspire Dashboard (great for debugging). Two `transform` statements give name parity with `/solr/admin/metrics`: clear the dimensionless unit `"1"` (the Collector would append `_ratio`), and replace `service.name=unknown_service:java` with `solr` so `job` stays `solr`. - **Dev stack**: `otel-collector` (contrib 0.164.0) and `aspire-dashboard` (13.6.2) services; bundled nodes always push metrics and traces. `./stack.sh --metrics-scrape=solr|otel` selects what Prometheus scrapes, job named `solr` either way, so the dashboard and rules are unchanged between modes. Default `solr`, overridable via `METRICS_SCRAPE` in a gitignored `dev/.env`. Compose project renamed `dev` → `solr-telemetry-dev`; bundled nodes default to `apache/solr-nightly:10.2.0-SNAPSHOT`. - **`dev/check-otel-parity.sh`** (new): diffs metric names, types and label keys between a Solr node and the Collector, using Prometheus ingest semantics. How the config was validated, and how to re-validate after a Collector or OTel upgrade. - **Docs**: fills the `Option 2: OTLP Push` placeholder in the monitoring ref guide page; `metrics-reporting.adoc` points at the shipped file instead of its own ad-hoc config; README covers the artifact and the switch. ## Validation On `apache/solr-nightly:10.2.0-SNAPSHOT` (OTel 1.65.0), compared at the endpoint and again through Prometheus: **135 metric names via each route, `job="solr"` on both, identical name/type/label keys for every shared metric**. Remaining diffs are per-node role metrics (overseer and peer-sync on one node, sync-with-leader on the other) and `target_info`. `rat`, `shellcheck`, `promtool check config`, `otelcol validate`, ref-guide `check`: all pass. ## Known gaps - No per-node identity over OTLP: Solr's meter providers carry no `service.instance.id`, so `instance` is the Collector's address and no `target_info` is produced — panels grouping by `instance` show one merged series in `otel` mode. SOLR-18594 fixes it; the ref guide says so. Confirmed by inserting a `service.instance.id` at the Collector, which makes both appear. - Only the scrape-the-Collector route is verified. Pushing into Prometheus (`prometheusremotewrite`, or `otlphttp` into its OTLP receiver) is documented but not exercised, as Prometheus applies its own name translation. - Pre-existing, filed separately: enabling OTLP push makes `/solr/admin/metrics` return HTTP 500 (`DuplicateLabelsException`). It is why `--metrics-scrape=solr` shows the bundled nodes down on a current nightly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
