Copilot commented on code in PR #5244: URL: https://github.com/apache/solr/pull/5244#discussion_r4388775090
########## solr/monitoring/dev/docker-compose.yml: ########## @@ -42,24 +42,51 @@ # Prometheus http://localhost:9090 # Grafana http://localhost:3000 (admin / admin) # Alertmanager http://localhost:9093 +# otel-collector OTLP in on http://localhost:4318, Prometheus-format out on http://localhost:9464 +# Aspire http://localhost:18888 (raw view of the metrics and traces Solr pushed) # trafficgen (no port — continuously indexes/searches the 'test' collection) # +# The bundled Solr nodes always push metrics and traces over OTLP to otel-collector, +# which forwards both to the Aspire Dashboard. Prometheus scrapes either the nodes +# directly or the collector, depending on ./stack.sh --metrics-scrape=solr|otel. +# See ../otel-collector-config.yaml. +# # Environment variables (set by stack.sh, or override manually): # PROMETHEUS_CONFIG Path to the prometheus config to mount (default: ./prometheus/prometheus.yml) +# METRICS_SCRAPE solr (default) or otel — read by stack.sh, also from a ./.env file # SOLR_URL Solr base URL for trafficgen (default: http://solr1:8983/solr) -# SOLR_IMAGE Solr image for the bundled nodes (default: solr:10) +# SOLR_IMAGE Solr image for the bundled nodes (default: apache/solr-nightly:10.2.0-SNAPSHOT, tracks branch_10x) # PROMETHEUS_IMAGE Prometheus image (default: pinned, see below) # GRAFANA_IMAGE Grafana image (default: pinned, see below) # ALERTMANAGER_IMAGE Alertmanager image (default: pinned, see below) +# OTEL_COLLECTOR_IMAGE OpenTelemetry Collector image (default: pinned, see below; must be a contrib build) +# ASPIRE_IMAGE Aspire Dashboard image (default: pinned, see below) # # The monitoring images are pinned so a stack that worked yesterday still works # today; override any of them to test against a newer release. # ============================================================================ +name: solr-telemetry-dev + +# Push metrics to the collector every 15s, matching the scrape interval. The +# endpoint is the OTLP/HTTP default with the host swapped for the service name. +# OTEL_SERVICE_NAME activates the module's tracer, which pushes traces the same way. +x-solr-otlp-env: &solr-otlp-env + SOLR_MODULES: opentelemetry + SOLR_METRICS_OTLP_EXPORTER_ENABLED: "true" Review Comment: Enabling OTLP metrics unconditionally also enables it in the default `--metrics-scrape=solr` mode. The PR's validation notes that this makes `/solr/admin/metrics` return HTTP 500 on the selected nightly, so a plain `./stack.sh` now loses its Solr metrics and fires `SolrNodeDown` for otherwise healthy nodes. Please keep OTLP metrics disabled in direct-scrape mode until that failure is fixed, or include the prerequisite fix before enabling it by default. ########## solr/monitoring/dev/check-otel-parity.sh: ########## @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# check-otel-parity.sh — verify that the OTel collector re-exposes the same Prometheus +# metric names, types and label keys as Solr's own metrics endpoint. +# +# Usage: +# ./check-otel-parity.sh [SOLR_METRICS_URL] [COLLECTOR_METRICS_URL] +# +# Defaults: http://localhost:8983/solr/admin/metrics and http://localhost:9464/metrics +# (the bundled solr1 and otel-collector of this stack). Exit code 1 on any difference. +# +# Both endpoints are fetched with the Accept header Prometheus sends, and compared the +# way Prometheus ingests them: empty-valued labels are dropped, and the job/instance +# labels are ignored since Prometheus sets those itself on a direct scrape. Metrics +# that only exist on one side because the event never happened there (e.g. no errors +# yet) show up as differences; let the trafficgen run a minute first. + +set -euo pipefail + +SOLR_URL="${1:-http://localhost:8983/solr/admin/metrics}"; +COLLECTOR_URL="${2:-http://localhost:9464/metrics}"; +ACCEPT='application/openmetrics-text;version=1.0.0;q=0.5,application/openmetrics-text;version=0.0.1;q=0.4,text/plain;version=1.0.0;q=0.3,text/plain;version=0.0.4;q=0.2,*/*;q=0.1' + +tmp="$(mktemp -d)" +trap 'rm -rf "$tmp"' EXIT + +curl -sf -H "Accept: $ACCEPT" "$SOLR_URL" > "$tmp/solr.txt" +curl -sf -H "Accept: $ACCEPT" "$COLLECTOR_URL" > "$tmp/collector.txt" + +# One line per series: <name> <sorted label keys>, plus one per family: TYPE <name> <type> +signature() { + grep -v '^#' "$1" | sed -E 's/^([a-zA-Z_:][a-zA-Z0-9_:]*)(\{([^}]*)\})?.*/\1 \3/' \ + | awk '{ n=split($2,kv,","); keys="" Review Comment: Using AWK's `$2` truncates the label list at the first space inside a quoted value, such as `jvm_memory_pool_name="G1 Eden Space"`. Removing subsequent labels such as `jvm_memory_type` then produces the same signature, allowing the script to report `Parity OK` despite different label keys. Parse the complete label list with quote- and escape-aware handling, then sort its keys before comparing. ########## solr/monitoring/dev/prometheus/prometheus-otel.yml: ########## @@ -0,0 +1,45 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# prometheus-otel.yml — scrape config for the OTLP route. +# +# Used by: ./stack.sh --metrics-scrape=otel +# +# Solr pushes metrics over OTLP to the otel-collector container, which re-exposes +# them in Prometheus format on :9464. The job is still named "solr" so the +# dashboard and alert rules apply unchanged. Works for bundled and local Solr alike. Review Comment: The shipped alert does not retain its meaning with this scrape target: `SolrNodeDown` uses `up{job="solr"} == 0`, and here `up` measures the Collector, not any Solr node. Stopping every Solr node while leaving the Collector running therefore never fires that alert. Please qualify this claim and document the limitation in the OTLP setup, or retain a separate per-node health scrape and scope the node-down rule to it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
