[
https://issues.apache.org/jira/browse/HDDS-17037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17037:
------------------------------
Status: Patch Available (was: Open)
> Intermittent failure in TestReconAndAdminContainerCLI init with Rocks
> Database is closed
> ----------------------------------------------------------------------------------------
>
> Key: HDDS-17037
> URL: https://issues.apache.org/jira/browse/HDDS-17037
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Recon, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI -- Time elapsed:
> 34.91 s <<< ERROR!
> java.lang.RuntimeException:
> org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Rocks Database is
> closed
> at
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.lambda$setupRatisKey$0(TestReconAndAdminContainerCLI.java:427)
> at
> org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:131)
> at
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.setupRatisKey(TestReconAndAdminContainerCLI.java:422)
> at
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.init(TestReconAndAdminContainerCLI.java:185)
> Caused by: org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Rocks
> Database is closed
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.acquire(RocksDatabase.java:417)
> ...
> at
> org.apache.hadoop.ozone.recon.spi.impl.ReconContainerMetadataManagerImpl.getKeyCountForContainer(ReconContainerMetadataManagerImpl.java:282)
> at
> org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.lambda$setupRatisKey$0(TestReconAndAdminContainerCLI.java:425)
> {code}
> h3. Root cause
> After the full OM snapshot, Recon reprocesses into a staging DB and swaps it
> in. {{ReconDBProvider#replaceStagedDb}} closes the current DB before
> {{ReconTaskControllerImpl}} calls
> {{ReconContainerMetadataManager#reinitialize}} to point the tables at the new
> one. A read in that window fails with {{RocksDatabaseException: Rocks
> Database is closed}}. {{setupRatisKey}} polls {{getKeyCountForContainer}} and
> rethrows any {{IOException}} as {{RuntimeException}}, so the transient error
> fails {{@BeforeAll}} instead of being retried.
> h3. Fix
> Use the existing {{ReconOmMetaManagerTestUtils#waitUntilReconKeyCounts}},
> which treats such {{IOException}} as not ready yet and keeps polling.
> - https://github.com/apache/ozone/actions/runs/38652082329/job/116157582872
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
