Siyao Meng created HDDS-17037:
---------------------------------
Summary: Intermittent failure in TestReconAndAdminContainerCLI
init with Rocks Database is closed
Key: HDDS-17037
URL: https://issues.apache.org/jira/browse/HDDS-17037
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Recon, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code}
org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI -- Time elapsed:
34.91 s <<< ERROR!
java.lang.RuntimeException:
org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Rocks Database is closed
at
org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.lambda$setupRatisKey$0(TestReconAndAdminContainerCLI.java:427)
at
org.apache.ozone.test.GenericTestUtils.waitFor(GenericTestUtils.java:131)
at
org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.setupRatisKey(TestReconAndAdminContainerCLI.java:422)
at
org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.init(TestReconAndAdminContainerCLI.java:185)
Caused by: org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Rocks
Database is closed
at
org.apache.hadoop.hdds.utils.db.RocksDatabase.acquire(RocksDatabase.java:417)
...
at
org.apache.hadoop.ozone.recon.spi.impl.ReconContainerMetadataManagerImpl.getKeyCountForContainer(ReconContainerMetadataManagerImpl.java:282)
at
org.apache.hadoop.ozone.recon.TestReconAndAdminContainerCLI.lambda$setupRatisKey$0(TestReconAndAdminContainerCLI.java:425)
{code}
h3. Root cause
After the full OM snapshot, Recon reprocesses into a staging DB and swaps it
in. {{ReconDBProvider#replaceStagedDb}} closes the current DB before
{{ReconTaskControllerImpl}} calls
{{ReconContainerMetadataManager#reinitialize}} to point the tables at the new
one. A read in that window fails with {{RocksDatabaseException: Rocks Database
is closed}}. {{setupRatisKey}} polls {{getKeyCountForContainer}} and rethrows
any {{IOException}} as {{RuntimeException}}, so the transient error fails
{{@BeforeAll}} instead of being retried.
h3. Fix
Use the existing {{ReconOmMetaManagerTestUtils#waitUntilReconKeyCounts}}, which
treats such {{IOException}} as not ready yet and keeps polling.
- https://github.com/apache/ozone/actions/runs/38652082329/job/116157582872
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
