Wei-Chiu Chuang created HDDS-16534:
--------------------------------------
Summary: [Docs] Bootstrap a faulty OM
Key: HDDS-16534
URL: https://issues.apache.org/jira/browse/HDDS-16534
Project: Apache Ozone
Issue Type: Sub-task
Components: documentation
Reporter: Wei-Chiu Chuang
If an OM fails to start due to corrupt OM db, or corrupt Ratis logs, it is
possible to bootstrap its metadata from scratch.
Let's say OM fails with this error message (HDDS-15068):
{noformat}
2026-09-21 21:49:56,340 ERROR
[main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with
exception
java.util.concurrent.CompletionException: java.lang.IllegalStateException: gap
between start index 92074675 and first entry to append 131115267
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at
java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
at
java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
at
org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840){noformat}
Steps:
1. Stop the faulty OM if it's not stopped yet.
2. Take a back up of Ozone Manager Metadata Directory (ozone.om.db.dirs
) e.g. /var/lib/hadoop-ozone/om/data and delete it.
3. If the OM db is very large, apply the tunings
ozone.om.ratis.log.purge.preservation.log.num to a high value (e.g. 1000000)
and increase ozone.om.snapshot.checkpoint.dir.creation.poll.timeout to 1
minute. (Described in details in Troubleshooting OM HA snapshot installation
issues)
4. Restart the other 2 healthy OM as well to apply the change.
5. Once the other 2 OMs are up and healthy, restart the faulty OM.
The faulty OM should download the OM db checkpoint, load it up and start from
there.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
