[
https://issues.apache.org/jira/browse/SOLR-18595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SOLR-18595:
----------------------------------
Labels: pull-request-available (was: )
> Metrics endpoint returns HTTP 500 (DuplicateLabelsException) when OTLP
> metrics push is enabled
> ----------------------------------------------------------------------------------------------
>
> Key: SOLR-18595
> URL: https://issues.apache.org/jira/browse/SOLR-18595
> Project: Solr
> Issue Type: Bug
> Components: metrics
> Affects Versions: 10.1
> Reporter: Jan Høydahl
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> With the {{opentelemetry}} module enabled and
> {{{}solr.metrics.otlpExporterEnabled=true{}}}, both {{/solr/admin/metrics}}
> and {{/api/metrics}} answer *HTTP 500* for the entire response:
> {code:java}
> io.prometheus.metrics.model.snapshots.DuplicateLabelsException: Duplicate
> labels for metric
> "otel_sdk_metric_reader_collection_duration_seconds":
> {otel_component_name="periodic_metric_reader/0",otel_component_type="periodic_metric_reader",otel_scope_name="io.opentelemetry.sdk.metrics"}
> {code}
> So pull and push cannot be used together: enabling push takes the scrape
> endpoint down, Prometheus marks every node as failing, and the Admin UI
> metrics views break. Nothing in the response points at the exporter as the
> cause.
> h2. Root cause
> {{SolrMetricManager.meterProvider()}} builds one {{SdkMeterProvider}} per
> registry (node, jvm, one per core). With an OTLP exporter configured, each
> gets its own {{PeriodicMetricReader}} next to its
> {{{}FilterablePrometheusMetricReader{}}}.
> Since OpenTelemetry 1.57 that reader records
> {{otel.sdk.metric_reader.collection.duration}} into the provider it is
> registered to, identified by a *static* field:
> {code:java}
> // io/opentelemetry/sdk/metrics/export/PeriodicMetricReader.java (1.65.0)
> private static final ComponentId COMPONENT_ID =
> ComponentId.generateLazy("periodic_metric_reader");
> {code}
> Being static, every reader instance in the JVM shares one component name, so
> each registry emits that histogram with an identical label set.
> {{MetricsHandler}} merges all registries by Prometheus name through
> {{{}MetricUtils.mergeSnapshots{}}}, handing the Prometheus client several
> data points with the same labels, which it rejects – taking the whole
> snapshot set with it.
> Without the exporter no {{PeriodicMetricReader}} exists, which is why the
> endpoint is healthy by default.
> h2. Affected versions
> OpenTelemetry 1.57+, i.e. 10.1 and later ({{{}branch_10_1{}}}, {{branch_10x}}
> and {{main}} are on 1.65.0). 10.0.0 shipped 1.56.0 and is unaffected.
> h2. Reproducing
> {code:java}
> SOLR_MODULES=opentelemetry SOLR_METRICS_OTLP_EXPORTER_ENABLED=true bin/solr
> start
> curl -i 'http://localhost:8983/solr/admin/metrics'
> {code}
> Nothing need be listening on the OTLP endpoint; the duration is recorded
> whether or not the export succeeds.
> h2. Candidate fix
> Suppress self-telemetry Solr never meant to expose:
> {code:java}
> PeriodicMetricReader.builder(metricExporter)
> .setInterval(OTLP_EXPORTER_INTERVAL, TimeUnit.MILLISECONDS)
> .setInternalTelemetryVersion(InternalTelemetryVersion.LEGACY)
> .build()
> {code}
> With {{{}LEGACY{}}}, {{PeriodicMetricReader.setMeterProvider()}} skips
> creating the instrumentation entirely. Verified by reading the 1.65.0 source,
> not yet at runtime.
> Worth considering alongside, as a guard against the next SDK component that
> does this: have {{MetricUtils.mergeSnapshots}} drop duplicates with a
> warning, or fail per metric family instead of failing the whole response.
> h2. Test gap
> Nothing covers the combination: {{SolrMetricManagerTest}} registers an
> {{InMemoryMetricExporter}} but never renders Prometheus output;
> {{TestPrometheusResponseWriter}} and {{TestPrometheusResponseWriterCloud}}
> render output but configure no exporter. A regression test should assert HTTP
> 200 with an exporter registered and several registries populated.
>
> ---
> Analysis and bug description done by 🤖 Claude Code 🤖
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
