[
https://issues.apache.org/jira/browse/SOLR-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133225#comment-18133225
]
ASF subversion and git services commented on SOLR-18544:
--------------------------------------------------------
Commit a336bf871b2c2b67abe55808966051d1c553b579 in solr's branch
refs/heads/branch_10x from Matthew Biscocho
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=a336bf871b2 ]
SOLR-18544: Source CPU utilization metric from JFR instead JMX (#5185)
Partially a regression from SOLR-18407 but also partially a JDK bug that got
exposed from the OTEL upgrade.
This was because previously the metric was sourced from JFR. Then after the
upgrade it started to source it from JMX which failed to some JDK bug. So this
change just sources it from JFR again.
> PlacementPluginIntegrationTest.testAttributeFetcherImpl fails intermittently:
> jvm_system_cpu_utilization absent when getCpuLoad() returns NaN
> ---------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-18544
> URL: https://issues.apache.org/jira/browse/SOLR-18544
> Project: Solr
> Issue Type: Bug
> Components: metrics, Tests
> Affects Versions: 10.0, 10.1
> Reporter: David Smiley
> Priority: Major
> Labels: flaky-test, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> testAttributeFetcherImpl fails with "systemLoadAverage 2". It failed during
> the 10.1.0 RC smoke test on macOS and reproduces on branch_10_1 with:
> {noformat}
> ./gradlew :solr:core:test --tests
> "org.apache.solr.cluster.placement.impl.PlacementPluginIntegrationTest.testAttributeFetcherImpl"
> -Ptests.seed=FE4CEBEDABF758B2
> {noformat}
> *Cause*
> NodeMetricImpl.SYSLOAD_AVG maps to jvm_system_cpu_utilization. That metric is
> emitted by OTel runtime-telemetry's JMX SystemCpu gauge, which only records a
> value when OperatingSystemMXBean.getCpuLoad() is >= 0. On macOS, getCpuLoad()
> returns NaN when it is called again before the host CPU tick counters
> advance. A standalone probe showed:
> * 199,999 of 200,000 back-to-back calls returned NaN
> * 3 threads calling concurrently (like 3 nodes in one test JVM scraping
> metrics): 595 of 600 calls returned NaN
> So any node whose scrape happens right after another call reports no
> jvm_system_cpu_utilization, and the assertion fails. Other consumers of this
> metric (CPUCircuitBreaker, SolrJ NodeValueFetcher) read the same gauge. I
> have not checked whether Linux can return NaN this way.
> PR #4711 (metric rename) addressed earlier failures of this test, but not
> this case.
> *Possible fixes*
> * Test: tolerate an absent SYSLOAD_AVG, or retry the fetch until present.
> * Production: have OtelRuntimeJvmMetrics prefer the JFR source
> (OverallCpuLoadHandler, jdk.CPULoad event) for jvm.system.cpu.utilization. It
> always records a value (updated every 1s, initially 0).
> 🤖 This issue was drafted with AI assistance (Claude Code).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
