Siyao Meng created HDDS-16119:
---------------------------------
Summary: DatanodeStorageMetrics leaks a metrics source and its
MutableVolumeSet per datanode in mini-cluster JVMs
Key: HDDS-16119
URL: https://issues.apache.org/jira/browse/HDDS-16119
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
{{DatanodeStorageMetrics}} (added in HDDS-13128) registers with the metrics
system under a constant source name and keeps a {{final MutableVolumeSet}}
reference. {{OzoneContainer}} creates one instance per datanode.
Integration tests run many datanodes in a single JVM and enable
{{DefaultMetricsSystem.setMiniClusterMode(true)}} (see {{MiniOzoneClusterImpl}}
and {{MiniOzoneHAClusterImpl}}). In mini-cluster mode the metrics system
uniquifies duplicate source names, so the second and later datanodes register
as {{DatanodeStorageMetrics-1}}, {{DatanodeStorageMetrics-2}}, and so on.
However {{DatanodeStorageMetrics.unregister()}} calls
{{unregisterSource("DatanodeStorageMetrics")}} using the constant base name, so
only the first datanode's source is ever removed. Every datanode beyond the
first leaks its metrics source, and because each leaked source holds a
{{MutableVolumeSet}}, the entire volume set graph of the shut-down datanode
stays reachable and cannot be collected.
Within any single test class that stands up multiple datanodes (or builds
clusters back to back), the leaked sources and their pinned volume sets
accumulate in that class's JVM until the class finishes. This is a correctness
and heap-hygiene bug: shut-down datanode state that should be collectable stays
reachable, and stale {{DatanodeStorageMetrics-N}} sources pile up in the
metrics registry.
The older {{ContainerMetrics}} uses a similar constant-name register and
unregister and also leaks its source in mini-cluster mode, but it retains only
lightweight counters, so it was tolerated. {{DatanodeStorageMetrics}} is
qualitatively worse because it pins a full datanode volume set.
h3. Reproduction (unit level, no cluster)
With {{DefaultMetricsSystem.setMiniClusterMode(true)}}, create N
{{DatanodeStorageMetrics}} over real {{MutableVolumeSet}} instances, shut the
volume sets down, then call {{unregister()}} on each (as
{{OzoneContainer.stop()}} does), and force GC. Volume sets held only through
the leaked metrics sources remain reachable. Measured with 15 clusters of 3
datanodes: 30 of 45 volume sets survived GC with {{DatanodeStorageMetrics}},
versus 0 of 45 for the pre-change baseline.
h3. Proposed fix
Register and unregister under a per-instance unique source name, following the
existing {{VolumeInfoMetrics}} pattern ({{SOURCE_BASENAME + '-' +
identifier}}). This keeps register and unregister symmetric so no source or
volume set is leaked. With the fix the same reproduction retains 0 of 45 volume
sets.
h3. Testing
Unit test that asserts, in mini-cluster mode, that repeated create plus
unregister cycles leave no residual {{DatanodeStorageMetrics}} sources and that
shut-down {{MutableVolumeSet}} instances become unreachable after GC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
