janhoy commented on code in PR #5244: URL: https://github.com/apache/solr/pull/5244#discussion_r4410410293
########## solr/monitoring/dev/docker-compose.yml: ########## @@ -42,24 +42,51 @@ # Prometheus http://localhost:9090 # Grafana http://localhost:3000 (admin / admin) # Alertmanager http://localhost:9093 +# otel-collector OTLP in on http://localhost:4318, Prometheus-format out on http://localhost:9464 +# Aspire http://localhost:18888 (raw view of the metrics and traces Solr pushed) # trafficgen (no port — continuously indexes/searches the 'test' collection) # +# The bundled Solr nodes always push metrics and traces over OTLP to otel-collector, +# which forwards both to the Aspire Dashboard. Prometheus scrapes either the nodes +# directly or the collector, depending on ./stack.sh --metrics-scrape=solr|otel. +# See ../otel-collector-config.yaml. +# # Environment variables (set by stack.sh, or override manually): # PROMETHEUS_CONFIG Path to the prometheus config to mount (default: ./prometheus/prometheus.yml) +# METRICS_SCRAPE solr (default) or otel — read by stack.sh, also from a ./.env file # SOLR_URL Solr base URL for trafficgen (default: http://solr1:8983/solr) -# SOLR_IMAGE Solr image for the bundled nodes (default: solr:10) +# SOLR_IMAGE Solr image for the bundled nodes (default: apache/solr-nightly:10.2.0-SNAPSHOT, tracks branch_10x) # PROMETHEUS_IMAGE Prometheus image (default: pinned, see below) # GRAFANA_IMAGE Grafana image (default: pinned, see below) # ALERTMANAGER_IMAGE Alertmanager image (default: pinned, see below) +# OTEL_COLLECTOR_IMAGE OpenTelemetry Collector image (default: pinned, see below; must be a contrib build) +# ASPIRE_IMAGE Aspire Dashboard image (default: pinned, see below) # # The monitoring images are pinned so a stack that worked yesterday still works # today; override any of them to test against a newer release. # ============================================================================ +name: solr-telemetry-dev + +# Push metrics to the collector every 15s, matching the scrape interval. The +# endpoint is the OTLP/HTTP default with the host swapped for the service name. +# OTEL_SERVICE_NAME activates the module's tracer, which pushes traces the same way. +x-solr-otlp-env: &solr-otlp-env + SOLR_MODULES: opentelemetry + SOLR_METRICS_OTLP_EXPORTER_ENABLED: "true" Review Comment: Right about the symptom, but the prerequisite exists rather than being absent: the HTTP 500 is SOLR-18595, fixed in #5247, and this PR is sequenced to land after it. I have made that explicit at the top of the description. So I am keeping the push enabled in both modes — that is what gives the Collector and Aspire data in either mode, and it is correct once #5247 is in. The only residual window is that `SOLR_IMAGE` defaults to a nightly, so the fix is not in the image until a nightly carrying it is published; that is now a `TODO SOLR-18595` in `docker-compose.yml` naming the two workarounds (a locally built image, or `--metrics-scrape=otel`), plus a note in the README. ########## solr/monitoring/dev/prometheus/prometheus-otel.yml: ########## @@ -0,0 +1,45 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# prometheus-otel.yml — scrape config for the OTLP route. +# +# Used by: ./stack.sh --metrics-scrape=otel +# +# Solr pushes metrics over OTLP to the otel-collector container, which re-exposes +# them in Prometheus format on :9464. The job is still named "solr" so the +# dashboard and alert rules apply unchanged. Works for bundled and local Solr alike. Review Comment: Agreed, and the claim was too broad. Under push Prometheus only scrapes the Collector, so `up{job="solr"}` is the Collector and `SolrNodeDown` reports its reachability; a node that stops pushing fires nothing, and a Collector restart alerts as though every node were down. I documented it rather than restructuring the scrape: the shipped rule is `up{job="solr"} == 0`, so a separate per-node health job would either need a different job name the rule does not match or a mixin change, which belongs with the real alert rules in SOLR-18535. There is now a WARNING in the `Option 2: OTLP Push` section spelling out the three failure modes and pointing at the two ways to keep node-down detection (retain a direct scrape, or alert per-node once SOLR-18594 gives the nodes distinct `instance` labels), and the same caveat in `prometheus-otel.yml`. I also softened its "alert rules apply unchanged" line, which contradicted it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
