[
https://issues.apache.org/jira/browse/HDDS-17501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18133214#comment-18133214
]
Sergey Soldatov commented on HDDS-17501:
----------------------------------------
Seen in CI:
https://github.com/ss77892/ozone/actions/runs/40553006606/job/122375507065
TestBackgroundContainerDataScannerIntegration>ContainerScannerIntegrationTests.shutdown:116
java.lang.AssertionError: Found 1 metrics source(s) still registered after
cluster shutdown: [RDBMetrics]
Root cause: {{OzoneContainer#stopContainerScrub}} returns early when
{{metadataScanner == null}}. Since HDDS-8553, the metadata scanner is optional
({{hdds.container.scrub.metadata.enabled}}). When it is disabled, the
background data scanners and the on-demand scanner are never shut down. The
test hits this now because
HDDS-17301 fixed the config key, so {{setMetadataScanEnabled(false)}}
actually takes effect.
How the leak happens:
{{volumeSet.shutdown()}} closes the per-volume RocksDB. {{RDBStore#close}}
calls {{RDBMetrics.unRegister()}}, which unregisters the source shared by all
stores in the JVM.
A data scanner that is still running calls {{BlockUtils.getDB()}}. That
reopens the volume DB through {{DatanodeStoreCache}}, which registers
{{RDBMetrics}} again.
In mini cluster mode, {{DatanodeStoreCache#shutdownCache}} keeps its entries.
The reopened DB is never closed, and {{MetricsLeakAssertion}} reports
{{RDBMetrics}}.
The failure is intermittent because a scanner has to be scanning a container
just after the volumes are closed.
> OzoneContainer#stopContainerScrub do not stop all scanners
> ----------------------------------------------------------
>
> Key: HDDS-17501
> URL: https://issues.apache.org/jira/browse/HDDS-17501
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Ivan Andika
> Assignee: Jheng-Sing Chen
> Priority: Minor
>
> See https://github.com/apache/ozone/pull/11773#discussion_r4379237308
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
