[
https://issues.apache.org/jira/browse/HDDS-17038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Huang reassigned HDDS-17038:
---------------------------------
Assignee: Andy Huang
> OzoneManagerDoubleBuffer metrics disappear permanently after a follower
> installs a Ratis snapshot
> -------------------------------------------------------------------------------------------------
>
> Key: HDDS-17038
> URL: https://issues.apache.org/jira/browse/HDDS-17038
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM HA
> Affects Versions: 2.0.0, 2.2.0, 2.2.1
> Reporter: Aleksei Ieshin
> Assignee: Andy Huang
> Priority: Minor
>
> After an OM installs a Ratis snapshot (for example, a lagging follower
> catching up), all {{OzoneManagerDoubleBufferMetrics}}
> ({{{}ozone_manager_double_buffer_metrics_*{}}} on {{{}/prom{}}},
> {{OzoneManagerDoubleBufferMetrics}} on {{{}/jmx{}}}) stop being published.
> They stay missing until the OM JVM restarts. The double buffer itself keeps
> flushing; only the metrics source is gone.
> Cause:
> # {{{}OzoneManagerStateMachine.pause(){}}}, used by snapshot install, calls
> {{{}ozoneManagerDoubleBuffer.stop(){}}}. {{OzoneManagerDoubleBuffer.stop()}}
> calls {{{}metrics.unRegister(){}}}, which runs
> {{{}DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME){}}}.
> # {{OzoneManagerStateMachine.unpause()}} then creates a new double buffer
> via {{{}buildDoubleBufferForRatis(){}}}. Its field initializer calls
> {{{}OzoneManagerDoubleBufferMetrics.create(){}}}.
> # {{create()}} is a lazy singleton:
>
> !http://localhost:63342/markdownPreview/651969378//Users/aieshin/git/ozone!
> {code:java}
> public static synchronized OzoneManagerDoubleBufferMetrics create() {
> if (instance != null) {
> return instance; // still the old object, whose source was
> unregistered in step 1
> } ... {code}
> {{`unRegister()`}} never resets {{{}instance{}}}, so the new double buffer
> updates an object that no longer belongs to the metrics system, and nothing
> is exported.
> Observed in production on 2.0.0. The same code is in 2.2.1 and master. On a
> 3-node OM HA cluster, two followers lost these metrics after snapshot
> installs (one during a RATIS-2240 snapshot-install deadlock, one during a
> period of heavy load). Both got them back only after their pods restarted.
> The leader, which never installs a snapshot, was unaffected. This silently
> blinds dashboards and alerts built on double-buffer flush time.
> *Proposed fix:* make {{unRegister()}} reset the singleton, so the next
> {{create()}} registers a fresh source:
>
> {code:java}
> public static synchronized void unRegister() { // or keep it an instance
> method, but clear the static
> DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME);
> instance = null;
> } {code}
> Alternatively, {{create()}} could re-register when {{instance != null}} but
> the source is missing from the metrics system.
> *Test:* in {{{}TestOzoneManagerStateMachine{}}}, or via
> {{{}MiniOzoneHAClusterImpl{}}}, call {{pause()}} and then {{{}unpause(){}}}.
> Assert that
> {{DefaultMetricsSystem.instance().getSource("OzoneManagerDoubleBufferMetrics")}}
> is non-null, and that {{getTotalNumOfFlushOperations()}} increases after a
> further write. Possibly related: HDDS-16908 (intermittent failure in
> {{TestOzoneManagerDoubleBufferWithDummyResponse}} metrics assertions), since
> tests share the static singleton.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
