Aleksei Ieshin created HDDS-17038:
-------------------------------------
Summary: OzoneManagerDoubleBuffer metrics disappear permanently
after a follower installs a Ratis snapshot
Key: HDDS-17038
URL: https://issues.apache.org/jira/browse/HDDS-17038
Project: Apache Ozone
Issue Type: Bug
Components: OM HA
Affects Versions: 2.2.1, 2.2.0, 2.0.0
Reporter: Aleksei Ieshin
After an OM installs a Ratis snapshot (for example, a lagging follower catching
up), all {{OzoneManagerDoubleBufferMetrics}}
({{{}ozone_manager_double_buffer_metrics_*{}}} on {{{}/prom{}}},
{{OzoneManagerDoubleBufferMetrics}} on {{{}/jmx{}}}) stop being published. They
stay missing until the OM JVM restarts. The double buffer itself keeps
flushing; only the metrics source is gone.
Cause:
# {{{}OzoneManagerStateMachine.pause(){}}}, used by snapshot install, calls
{{{}ozoneManagerDoubleBuffer.stop(){}}}. {{OzoneManagerDoubleBuffer.stop()}}
calls {{{}metrics.unRegister(){}}}, which runs
{{{}DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME){}}}.
# {{OzoneManagerStateMachine.unpause()}} then creates a new double buffer via
{{{}buildDoubleBufferForRatis(){}}}. Its field initializer calls
{{{}OzoneManagerDoubleBufferMetrics.create(){}}}.
# {{create()}} is a lazy singleton:
!http://localhost:63342/markdownPreview/651969378//Users/aieshin/git/ozone!
{{public static synchronized OzoneManagerDoubleBufferMetrics create() \{
if (instance != null) {
return instance; // still the old object, whose source was
unregistered in step 1
} ...}}
{{unRegister()}} never resets {{{}instance{}}}, so the new double buffer
updates an object that no longer belongs to the metrics system, and nothing is
exported.
Observed in production on 2.0.0. The same code is in 2.2.1 and master. On a
3-node OM HA cluster, two followers lost these metrics after snapshot installs
(one during a RATIS-2240 snapshot-install deadlock, one during a period of
heavy load). Both got them back only after their pods restarted. The leader,
which never installs a snapshot, was unaffected. This silently blinds
dashboards and alerts built on double-buffer flush time.
*Proposed fix:* make {{unRegister()}} reset the singleton, so the next
{{create()}} registers a fresh source:
!http://localhost:63342/markdownPreview/651969378//Users/aieshin/git/ozone!
{{public static synchronized void unRegister() \{ // or keep it an instance
method, but clear the static
DefaultMetricsSystem.instance().unregisterSource(SOURCE_NAME);
instance = null;
}}}
Alternatively, {{create()}} could re-register when {{instance != null}} but the
source is missing from the metrics system.
*Test:* in {{{}TestOzoneManagerStateMachine{}}}, or via
{{{}MiniOzoneHAClusterImpl{}}}, call {{pause()}} and then {{{}unpause(){}}}.
Assert that
{{DefaultMetricsSystem.instance().getSource("OzoneManagerDoubleBufferMetrics")}}
is non-null, and that {{getTotalNumOfFlushOperations()}} increases after a
further write. Possibly related: HDDS-16908 (intermittent failure in
{{TestOzoneManagerDoubleBufferWithDummyResponse}} metrics assertions), since
tests share the static singleton.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
