[
https://issues.apache.org/jira/browse/HDDS-17037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18130509#comment-18130509
]
Siyao Meng commented on HDDS-17037:
-----------------------------------
[Jira cleanup 2026-10-22] Resolving as Fixed. Recon initialization uses the
helper that retries transient DB-read failures.
Evidence:
* https://github.com/apache/ozone/pull/11548
*
https://github.com/apache/ozone/commit/1bc0787cf7f500f3ac538d7fcc880fc00179ac2d
*
https://github.com/apache/ozone/blob/bf3829e6c8ac69ae17585585703f91426dcdf6b3/hadoop-ozone/integration-test-recon/src/test/java/org/apache/hadoop/ozone/recon/TestReconAndAdminContainerCLI.java#L422
Feel free to reopen if further work is needed.
> Intermittent failure in TestReconAndAdminContainerCLI init with Rocks
> Database is closed
> ----------------------------------------------------------------------------------------
>
> Key: HDDS-17037
> URL: https://issues.apache.org/jira/browse/HDDS-17037
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Recon, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI -- Time elapsed:
> 34.91 s <<< ERROR!
> java.lang.RuntimeException:
> org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Rocks Database is
> closed
> at
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.lambda$setupRatisKey$0(TestReconAndAdminContainerCLI.java:427)
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:131)
> at
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.setupRatisKey(TestReconAndAdminContainerCLI.java:422)
> at
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.init(TestReconAndAdminContainerCLI.java:185)
> Caused by: org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Rocks
> Database is closed
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.acquire(RocksDatabase.java:417)
> ...
> at
> org.apache.hadoop.ozone.recon.spi.impl.ReconContainerMetadataManagerImpl.getKeyCountForContainer(ReconContainerMetadataManagerImpl.java:282)
> at
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.lambda$setupRatisKey$0(TestReconAndAdminContainerCLI.java:425)
> {code}
> h3. Root cause
> After the full OM snapshot, Recon reprocesses into a staging DB and swaps it
> in. {{ReconDBProvider#replaceStagedDb}} closes the current DB before
> {{ReconTaskControllerImpl}} calls
> {{ReconContainerMetadataManager#reinitialize}} to point the tables at the new
> one. A read in that window fails with {{RocksDatabaseException: Rocks
> Database is closed}}. {{setupRatisKey}} polls {{getKeyCountForContainer}} and
> rethrows any {{IOException}} as {{RuntimeException}}, so the transient error
> fails {{@BeforeAll}} instead of being retried.
> h3. Fix
> Use the existing {{ReconOmMetaManagerTestUtils#waitUntilReconKeyCounts}},
> which treats such {{IOException}} as not ready yet and keeps polling.
> - https://github.com/apache/ozone/actions/runs/38652082329/job/116157582872
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
