[
https://issues.apache.org/jira/browse/SOLR-18407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133196#comment-18133196
]
ASF subversion and git services commented on SOLR-18407:
--------------------------------------------------------
Commit 5f5d1db8f7126a3f62b6602260e2a8f6f7200b45 in solr's branch
refs/heads/main from Matthew Biscocho
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=5f5d1db8f71 ]
SOLR-18544: Source CPU utilization metric from JFR instead JMX (#5185)
Partially a regression from SOLR-18407 but also partially a JDK bug that got
exposed from the OTEL upgrade.
This was because previously the metric was sourced from JFR. Then after the
upgrade it started to source it from JMX which failed to some JDK bug. So this
change just sources it from JFR again.
> Solr 10 emits duplicate JVM metrics from both runtime-telemetry-java8 and
> runtime-telemetry-java17 scopes on Java 17
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18407
> URL: https://issues.apache.org/jira/browse/SOLR-18407
> Project: Solr
> Issue Type: Bug
> Components: metrics
> Affects Versions: 10.0
> Reporter: Jerry Chung
> Assignee: Eric Pugh
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 10.1
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> When running Solr 10 on Java 17, the {{/solr/admin/metrics?wt=prometheus}}
> endpoint emits JVM metrics (e.g. {{{}jvm_gc_duration_seconds_count{}}},
> {{{}jvm_memory_used_bytes{}}}) from two OTEL instrumentation scopes
> simultaneously:
> * {{io.opentelemetry.runtime-telemetry-java17}} (JFR-based)
> * {{io.opentelemetry.runtime-telemetry-java8}} (JMX-based)
> {{OtelRuntimeJvmMetrics}} only calls {{.enableAllFeatures()}} without
> {{{}.enableExperimentalJmxTelemetry(){}}}, so the java8/JMX scope should not
> activate. However, the bundled version of
> {{opentelemetry-runtime-telemetry-java17}} ({{{}2.22.0-alpha{}}}) implicitly
> activates JMX metrics alongside JFR, producing duplicate series with
> conflicting values for the same pool (e.g. G1 Eden Space {{used}} differs
> between scopes by an order of magnitude).
> *Impact:* Metric consumers receive duplicate series with different values for
> the same metric, making it impossible to reliably use these metrics in
> dashboards or alerts without scope-specific filtering.
> *Expected behavior:* Only {{runtime-telemetry-java17}} (JFR) metrics should
> be emitted on Java 17, or there should be a configuration to select
> *Workaround:* None available without patching Solr or upgrading the bundled
> OTEL library to 2.13.x+ where the implicit JMX activation was fixed.
> *Related:* SOLR-17806 /
> [apache/solr#3494|https://github.com/apache/solr/issues/3494],
> opentelemetry-java-instrumentation#13332,
> opentelemetry-java-instrumentation#13337
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
