Jan Høydahl created SOLR-18595:
----------------------------------
Summary: Metrics endpoint returns HTTP 500
(DuplicateLabelsException) when OTLP metrics push is enabled
Key: SOLR-18595
URL: https://issues.apache.org/jira/browse/SOLR-18595
Project: Solr
Issue Type: Bug
Components: metrics
Reporter: Jan Høydahl
With the {{opentelemetry}} module enabled and
{{solr.metrics.otlpExporterEnabled=true}}, both {{/solr/admin/metrics}} and
{{/api/metrics}} answer *HTTP 500* for the entire response:
{code}
io.prometheus.metrics.model.snapshots.DuplicateLabelsException: Duplicate
labels for metric
"otel_sdk_metric_reader_collection_duration_seconds":
{otel_component_name="periodic_metric_reader/0",otel_component_type="periodic_metric_reader",otel_scope_name="io.opentelemetry.sdk.metrics"}
{code}
So pull and push cannot be used together: enabling push takes the scrape
endpoint down, Prometheus marks every node as failing, and the Admin UI metrics
views break. Nothing in the response points at the exporter as the cause.
h2. Root cause
{{SolrMetricManager.meterProvider()}} builds one {{SdkMeterProvider}} per
registry (node, jvm, one per core). With an OTLP exporter configured, each gets
its own {{PeriodicMetricReader}} next to its
{{FilterablePrometheusMetricReader}}.
Since OpenTelemetry 1.57 that reader records
{{otel.sdk.metric_reader.collection.duration}} into the provider it is
registered to, identified by a *static* field:
{code:java}
// io/opentelemetry/sdk/metrics/export/PeriodicMetricReader.java (1.65.0)
private static final ComponentId COMPONENT_ID =
ComponentId.generateLazy("periodic_metric_reader");
{code}
Being static, every reader instance in the JVM shares one component name, so
each registry emits that histogram with an identical label set.
{{MetricsHandler}} merges all registries by Prometheus name through
{{MetricUtils.mergeSnapshots}}, handing the Prometheus client several data
points with the same labels, which it rejects -- taking the whole snapshot set
with it.
Without the exporter no {{PeriodicMetricReader}} exists, which is why the
endpoint is healthy by default.
h2. Affected versions
OpenTelemetry 1.57+, i.e. 10.1 and later ({{branch_10_1}}, {{branch_10x}} and
{{main}} are on 1.65.0). 10.0.0 shipped 1.56.0 and is unaffected.
h2. Reproducing
{code}
SOLR_MODULES=opentelemetry SOLR_METRICS_OTLP_EXPORTER_ENABLED=true bin/solr
start
curl -i 'http://localhost:8983/solr/admin/metrics'
{code}
Nothing need be listening on the OTLP endpoint; the duration is recorded
whether or not the export succeeds.
h2. Candidate fix
Suppress self-telemetry Solr never meant to expose:
{code:java}
PeriodicMetricReader.builder(metricExporter)
.setInterval(OTLP_EXPORTER_INTERVAL, TimeUnit.MILLISECONDS)
.setInternalTelemetryVersion(InternalTelemetryVersion.LEGACY)
.build()
{code}
With {{LEGACY}}, {{PeriodicMetricReader.setMeterProvider()}} skips creating the
instrumentation entirely. Verified by reading the 1.65.0 source, not yet at
runtime.
Worth considering alongside, as a guard against the next SDK component that
does this: have {{MetricUtils.mergeSnapshots}} drop duplicates with a warning,
or fail per metric family instead of failing the whole response.
h2. Test gap
Nothing covers the combination: {{SolrMetricManagerTest}} registers an
{{InMemoryMetricExporter}} but never renders Prometheus output;
{{TestPrometheusResponseWriter}} and {{TestPrometheusResponseWriterCloud}}
render output but configure no exporter. A regression test should assert HTTP
200 with an exporter registered and several registries populated.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
