[
https://issues.apache.org/jira/browse/HDDS-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDDS-16534:
-----------------------------------
Description:
If an OM fails to start due to corrupt OM db, or corrupt Ratis logs, it is
possible to bootstrap its metadata from scratch.
Let's say OM fails with this error message (HDDS-15068):
{noformat}
2026-09-21 21:49:56,340 ERROR
[main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with
exception
java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap
between start index 92074675 and first entry to append 131115267
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at
java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
at
java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
at
org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840){noformat}
Steps:
1. Stop the faulty OM if it's not stopped yet.
2. Take a back up of Ozone Manager Metadata Directory (ozone.om.db.dirs
) e.g. /var/lib/hadoop-ozone/om/data and delete it.
3. If the OM db is very large, apply the tunings
ozone.om.ratis.log.purge.preservation.log.num to a high value (e.g. 1000000)
and increase ozone.om.snapshot.checkpoint.dir.creation.poll.timeout to 1
minute. (Described in details in Troubleshooting OM HA snapshot installation
issues)
4. Restart the other 2 healthy OM as well to apply the change.
5. Once the other 2 OMs are up and healthy, on the faulty OM, do
{code:java}
ozone om --init
{code} or copy the VERSION file from the backup directory to
${ozone.om.db.dirs}/om/current/VERSION
.
6. Restart the faulty OM. The faulty OM should download the OM db checkpoint,
load it up and start from there.
was:
If an OM fails to start due to corrupt OM db, or corrupt Ratis logs, it is
possible to bootstrap its metadata from scratch.
Let's say OM fails with this error message (HDDS-15068):
{noformat}
2026-09-21 21:49:56,340 ERROR
[main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with
exception
java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap
between start index 92074675 and first entry to append 131115267
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at
java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
at
java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
at
org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840){noformat}
Steps:
1. Stop the faulty OM if it's not stopped yet.
2. Take a back up of Ozone Manager Metadata Directory (ozone.om.db.dirs
) e.g. /var/lib/hadoop-ozone/om/data and delete it.
3. If the OM db is very large, apply the tunings
ozone.om.ratis.log.purge.preservation.log.num to a high value (e.g. 1000000)
and increase ozone.om.snapshot.checkpoint.dir.creation.poll.timeout to 1
minute. (Described in details in Troubleshooting OM HA snapshot installation
issues)
4. Restart the other 2 healthy OM as well to apply the change.
5. Once the other 2 OMs are up and healthy, on the faulty OM, do
{code:java}
ozone om --init
{code} or copy the VERSION file from the backup directory to
${ozone.om.db.dirs}/om/current/VERSION
.
The faulty OM should download the OM db checkpoint, load it up and start from
there.
> [Docs] Bootstrap a faulty OM
> ----------------------------
>
> Key: HDDS-16534
> URL: https://issues.apache.org/jira/browse/HDDS-16534
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: documentation
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> If an OM fails to start due to corrupt OM db, or corrupt Ratis logs, it is
> possible to bootstrap its metadata from scratch.
>
> Let's say OM fails with this error message (HDDS-15068):
>
> {noformat}
> 2026-09-21 21:49:56,340 ERROR
> [main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with
> exception
> java.util.concurrent.CompletionException: java.lang.IllegalStateException:
> gap between start index 92074675 and first entry to append 131115267
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> at
> java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
> at
> java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
> at
> org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:840){noformat}
>
> Steps:
> 1. Stop the faulty OM if it's not stopped yet.
> 2. Take a back up of Ozone Manager Metadata Directory (ozone.om.db.dirs
> ) e.g. /var/lib/hadoop-ozone/om/data and delete it.
> 3. If the OM db is very large, apply the tunings
> ozone.om.ratis.log.purge.preservation.log.num to a high value (e.g. 1000000)
> and increase ozone.om.snapshot.checkpoint.dir.creation.poll.timeout to 1
> minute. (Described in details in Troubleshooting OM HA snapshot installation
> issues)
> 4. Restart the other 2 healthy OM as well to apply the change.
> 5. Once the other 2 OMs are up and healthy, on the faulty OM, do
> {code:java}
> ozone om --init
> {code} or copy the VERSION file from the backup directory to
> ${ozone.om.db.dirs}/om/current/VERSION
> .
> 6. Restart the faulty OM. The faulty OM should download the OM db checkpoint,
> load it up and start from there.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
