Jyotirmoy Sinha created HDDS-13172:
--------------------------------------
Summary: OM down due to CompletionException
Key: HDDS-13172
URL: https://issues.apache.org/jira/browse/HDDS-13172
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Manager
Reporter: Jyotirmoy Sinha
Steps :
* Execute snapshot load workloads
* Workload executed for 3+ days
* Workload details -
** Volume - 100
** Buckets - 400 (4 per volume)
*** Ratis-fso
*** Ratis-obs
*** Ec-fso
*** Ec-obs
** Keys -
*** 500-1000 per snapshot
*** Size = 10kb
* Configs -
** ozone.compaction.service.enabled = True
** ozone.om.compaction.service.run.interval = 1m
** ozone.om.snapshot.compaction.dag.prune.daemon.run.interval = 5m
** ozone.scm.block.size = 1KB
OM Error logs -
{{}}
{code:java}
2025-06-03 05:22:20,972 INFO
[om1546335944-impl-thread1]-org.apache.ratis.server.storage.RaftStorageDirectory:
Lock on
/var/lib/hadoop-ozone/om/ratis/2b34c18d-ed7a-3124-99bd-ca13a35e4b88/in_use.lock
acquired by nodename [email protected]
2025-06-03 05:22:20,978 ERROR
[main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with
exception
java.util.concurrent.CompletionException: java.io.IOException: Failed to load
/var/lib/hadoop-ozone/om/ratis/2b34c18d-ed7a-3124-99bd-ca13a35e4b88/current/raft-meta
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at
java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
at
java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at
org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
at
org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.io.IOException: Failed to load
/var/lib/hadoop-ozone/om/ratis/2b34c18d-ed7a-3124-99bd-ca13a35e4b88/current/raft-meta
at
org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.load(RaftStorageMetadataFileImpl.java:118)
at
org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.lambda$getMetadata$0(RaftStorageMetadataFileImpl.java:55)
at
org.apache.ratis.util.ConcurrentUtils.lambda$updateAndGet$0(ConcurrentUtils.java:51)
at
java.base/java.util.concurrent.atomic.AtomicReference.updateAndGet(AtomicReference.java:210)
at
org.apache.ratis.util.ConcurrentUtils.updateAndGet(ConcurrentUtils.java:49)
at
org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getMetadata(RaftStorageMetadataFileImpl.java:55)
at
org.apache.ratis.server.storage.RaftStorageImpl.analyzeAndRecoverStorage(RaftStorageImpl.java:116)
at
org.apache.ratis.server.storage.RaftStorageImpl.initialize(RaftStorageImpl.java:63)
at
org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:169)
at
org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:338)
at
org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:188)
... 4 more
Caused by: java.io.IOException: Failed to parse 'term' from properties: {}
at
org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getTerm(RaftStorageMetadataFileImpl.java:96)
at
org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.load(RaftStorageMetadataFileImpl.java:116)
... 14 more
Caused by: java.io.IOException: 'term' not found in properties: {}
at
org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.lambda$getValue$2(RaftStorageMetadataFileImpl.java:89)
at java.base/java.util.Optional.orElseThrow(Optional.java:403)
at
org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getValue(RaftStorageMetadataFileImpl.java:88)
at
org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getTerm(RaftStorageMetadataFileImpl.java:94)
... 15 more
2025-06-03 05:22:20,982 INFO
[shutdown-hook-0]-org.apache.ranger.audit.provider.AuditProviderFactory: ==>
JVMShutdownHook.run() {code}
{{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
