[
https://issues.apache.org/jira/browse/SOLR-18595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Høydahl updated SOLR-18595:
-------------------------------
Description:
With the {{opentelemetry}} module enabled and
{{{}solr.metrics.otlpExporterEnabled=true{}}}, both {{/solr/admin/metrics}} and
{{/api/metrics}} answer *HTTP 500* for the entire response:
{code:java}
io.prometheus.metrics.model.snapshots.DuplicateLabelsException: Duplicate
labels for metric
"otel_sdk_metric_reader_collection_duration_seconds":
{otel_component_name="periodic_metric_reader/0",otel_component_type="periodic_metric_reader",otel_scope_name="io.opentelemetry.sdk.metrics"}
{code}
So pull and push cannot be used together: enabling push takes the scrape
endpoint down, Prometheus marks every node as failing, and the Admin UI metrics
views break. Nothing in the response points at the exporter as the cause.
h2. Root cause
{{SolrMetricManager.meterProvider()}} builds one {{SdkMeterProvider}} per
registry (node, jvm, one per core). With an OTLP exporter configured, each gets
its own {{PeriodicMetricReader}} next to its
{{{}FilterablePrometheusMetricReader{}}}.
Since OpenTelemetry 1.57 that reader records
{{otel.sdk.metric_reader.collection.duration}} into the provider it is
registered to, identified by a *static* field:
{code:java}
// io/opentelemetry/sdk/metrics/export/PeriodicMetricReader.java (1.65.0)
private static final ComponentId COMPONENT_ID =
ComponentId.generateLazy("periodic_metric_reader");
{code}
Being static, every reader instance in the JVM shares one component name, so
each registry emits that histogram with an identical label set.
{{MetricsHandler}} merges all registries by Prometheus name through
{{{}MetricUtils.mergeSnapshots{}}}, handing the Prometheus client several data
points with the same labels, which it rejects – taking the whole snapshot set
with it.
Without the exporter no {{PeriodicMetricReader}} exists, which is why the
endpoint is healthy by default.
h2. Affected versions
OpenTelemetry 1.57+, i.e. 10.1 and later ({{{}branch_10_1{}}}, {{branch_10x}}
and {{main}} are on 1.65.0). 10.0.0 shipped 1.56.0 and is unaffected.
h2. Reproducing
{code:java}
SOLR_MODULES=opentelemetry SOLR_METRICS_OTLP_EXPORTER_ENABLED=true bin/solr
start
curl -i 'http://localhost:8983/solr/admin/metrics'
{code}
Nothing need be listening on the OTLP endpoint; the duration is recorded
whether or not the export succeeds.
h2. Candidate fix
Suppress self-telemetry Solr never meant to expose:
{code:java}
PeriodicMetricReader.builder(metricExporter)
.setInterval(OTLP_EXPORTER_INTERVAL, TimeUnit.MILLISECONDS)
.setInternalTelemetryVersion(InternalTelemetryVersion.LEGACY)
.build()
{code}
With {{{}LEGACY{}}}, {{PeriodicMetricReader.setMeterProvider()}} skips creating
the instrumentation entirely. Verified by reading the 1.65.0 source, not yet at
runtime.
Worth considering alongside, as a guard against the next SDK component that
does this: have {{MetricUtils.mergeSnapshots}} drop duplicates with a warning,
or fail per metric family instead of failing the whole response.
h2. Test gap
Nothing covers the combination: {{SolrMetricManagerTest}} registers an
{{InMemoryMetricExporter}} but never renders Prometheus output;
{{TestPrometheusResponseWriter}} and {{TestPrometheusResponseWriterCloud}}
render output but configure no exporter. A regression test should assert HTTP
200 with an exporter registered and several registries populated.
---
Analysis and bug description done by 🤖 Claude Code 🤖
was:
With the {{opentelemetry}} module enabled and
{{solr.metrics.otlpExporterEnabled=true}}, both {{/solr/admin/metrics}} and
{{/api/metrics}} answer *HTTP 500* for the entire response:
{code}
io.prometheus.metrics.model.snapshots.DuplicateLabelsException: Duplicate
labels for metric
"otel_sdk_metric_reader_collection_duration_seconds":
{otel_component_name="periodic_metric_reader/0",otel_component_type="periodic_metric_reader",otel_scope_name="io.opentelemetry.sdk.metrics"}
{code}
So pull and push cannot be used together: enabling push takes the scrape
endpoint down, Prometheus marks every node as failing, and the Admin UI metrics
views break. Nothing in the response points at the exporter as the cause.
h2. Root cause
{{SolrMetricManager.meterProvider()}} builds one {{SdkMeterProvider}} per
registry (node, jvm, one per core). With an OTLP exporter configured, each gets
its own {{PeriodicMetricReader}} next to its
{{FilterablePrometheusMetricReader}}.
Since OpenTelemetry 1.57 that reader records
{{otel.sdk.metric_reader.collection.duration}} into the provider it is
registered to, identified by a *static* field:
{code:java}
// io/opentelemetry/sdk/metrics/export/PeriodicMetricReader.java (1.65.0)
private static final ComponentId COMPONENT_ID =
ComponentId.generateLazy("periodic_metric_reader");
{code}
Being static, every reader instance in the JVM shares one component name, so
each registry emits that histogram with an identical label set.
{{MetricsHandler}} merges all registries by Prometheus name through
{{MetricUtils.mergeSnapshots}}, handing the Prometheus client several data
points with the same labels, which it rejects -- taking the whole snapshot set
with it.
Without the exporter no {{PeriodicMetricReader}} exists, which is why the
endpoint is healthy by default.
h2. Affected versions
OpenTelemetry 1.57+, i.e. 10.1 and later ({{branch_10_1}}, {{branch_10x}} and
{{main}} are on 1.65.0). 10.0.0 shipped 1.56.0 and is unaffected.
h2. Reproducing
{code}
SOLR_MODULES=opentelemetry SOLR_METRICS_OTLP_EXPORTER_ENABLED=true bin/solr
start
curl -i 'http://localhost:8983/solr/admin/metrics'
{code}
Nothing need be listening on the OTLP endpoint; the duration is recorded
whether or not the export succeeds.
h2. Candidate fix
Suppress self-telemetry Solr never meant to expose:
{code:java}
PeriodicMetricReader.builder(metricExporter)
.setInterval(OTLP_EXPORTER_INTERVAL, TimeUnit.MILLISECONDS)
.setInternalTelemetryVersion(InternalTelemetryVersion.LEGACY)
.build()
{code}
With {{LEGACY}}, {{PeriodicMetricReader.setMeterProvider()}} skips creating the
instrumentation entirely. Verified by reading the 1.65.0 source, not yet at
runtime.
Worth considering alongside, as a guard against the next SDK component that
does this: have {{MetricUtils.mergeSnapshots}} drop duplicates with a warning,
or fail per metric family instead of failing the whole response.
h2. Test gap
Nothing covers the combination: {{SolrMetricManagerTest}} registers an
{{InMemoryMetricExporter}} but never renders Prometheus output;
{{TestPrometheusResponseWriter}} and {{TestPrometheusResponseWriterCloud}}
render output but configure no exporter. A regression test should assert HTTP
200 with an exporter registered and several registries populated.
> Metrics endpoint returns HTTP 500 (DuplicateLabelsException) when OTLP
> metrics push is enabled
> ----------------------------------------------------------------------------------------------
>
> Key: SOLR-18595
> URL: https://issues.apache.org/jira/browse/SOLR-18595
> Project: Solr
> Issue Type: Bug
> Components: metrics
> Reporter: Jan Høydahl
> Priority: Major
>
> With the {{opentelemetry}} module enabled and
> {{{}solr.metrics.otlpExporterEnabled=true{}}}, both {{/solr/admin/metrics}}
> and {{/api/metrics}} answer *HTTP 500* for the entire response:
> {code:java}
> io.prometheus.metrics.model.snapshots.DuplicateLabelsException: Duplicate
> labels for metric
> "otel_sdk_metric_reader_collection_duration_seconds":
> {otel_component_name="periodic_metric_reader/0",otel_component_type="periodic_metric_reader",otel_scope_name="io.opentelemetry.sdk.metrics"}
> {code}
> So pull and push cannot be used together: enabling push takes the scrape
> endpoint down, Prometheus marks every node as failing, and the Admin UI
> metrics views break. Nothing in the response points at the exporter as the
> cause.
> h2. Root cause
> {{SolrMetricManager.meterProvider()}} builds one {{SdkMeterProvider}} per
> registry (node, jvm, one per core). With an OTLP exporter configured, each
> gets its own {{PeriodicMetricReader}} next to its
> {{{}FilterablePrometheusMetricReader{}}}.
> Since OpenTelemetry 1.57 that reader records
> {{otel.sdk.metric_reader.collection.duration}} into the provider it is
> registered to, identified by a *static* field:
> {code:java}
> // io/opentelemetry/sdk/metrics/export/PeriodicMetricReader.java (1.65.0)
> private static final ComponentId COMPONENT_ID =
> ComponentId.generateLazy("periodic_metric_reader");
> {code}
> Being static, every reader instance in the JVM shares one component name, so
> each registry emits that histogram with an identical label set.
> {{MetricsHandler}} merges all registries by Prometheus name through
> {{{}MetricUtils.mergeSnapshots{}}}, handing the Prometheus client several
> data points with the same labels, which it rejects – taking the whole
> snapshot set with it.
> Without the exporter no {{PeriodicMetricReader}} exists, which is why the
> endpoint is healthy by default.
> h2. Affected versions
> OpenTelemetry 1.57+, i.e. 10.1 and later ({{{}branch_10_1{}}}, {{branch_10x}}
> and {{main}} are on 1.65.0). 10.0.0 shipped 1.56.0 and is unaffected.
> h2. Reproducing
> {code:java}
> SOLR_MODULES=opentelemetry SOLR_METRICS_OTLP_EXPORTER_ENABLED=true bin/solr
> start
> curl -i 'http://localhost:8983/solr/admin/metrics'
> {code}
> Nothing need be listening on the OTLP endpoint; the duration is recorded
> whether or not the export succeeds.
> h2. Candidate fix
> Suppress self-telemetry Solr never meant to expose:
> {code:java}
> PeriodicMetricReader.builder(metricExporter)
> .setInterval(OTLP_EXPORTER_INTERVAL, TimeUnit.MILLISECONDS)
> .setInternalTelemetryVersion(InternalTelemetryVersion.LEGACY)
> .build()
> {code}
> With {{{}LEGACY{}}}, {{PeriodicMetricReader.setMeterProvider()}} skips
> creating the instrumentation entirely. Verified by reading the 1.65.0 source,
> not yet at runtime.
> Worth considering alongside, as a guard against the next SDK component that
> does this: have {{MetricUtils.mergeSnapshots}} drop duplicates with a
> warning, or fail per metric family instead of failing the whole response.
> h2. Test gap
> Nothing covers the combination: {{SolrMetricManagerTest}} registers an
> {{InMemoryMetricExporter}} but never renders Prometheus output;
> {{TestPrometheusResponseWriter}} and {{TestPrometheusResponseWriterCloud}}
> render output but configure no exporter. A regression test should assert HTTP
> 200 with an exporter registered and several registries populated.
>
> ---
> Analysis and bug description done by 🤖 Claude Code 🤖
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
