[
https://issues.apache.org/jira/browse/HDDS-13172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jyotirmoy Sinha updated HDDS-13172:
-----------------------------------
Issue Type: Bug (was: Improvement)
> OM down due to CompletionException
> ----------------------------------
>
> Key: HDDS-13172
> URL: https://issues.apache.org/jira/browse/HDDS-13172
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Reporter: Jyotirmoy Sinha
> Priority: Major
>
> Steps :
> * Execute snapshot load workloads
> * Workload executed for 3+ days
> * Workload details -
> ** Volume - 100
> ** Buckets - 400 (4 per volume)
> *** Ratis-fso
> *** Ratis-obs
> *** Ec-fso
> *** Ec-obs
> ** Keys -
> *** 500-1000 per snapshot
> *** Size = 10kb
> * Configs -
> ** ozone.compaction.service.enabled = True
> ** ozone.om.compaction.service.run.interval = 1m
> ** ozone.om.snapshot.compaction.dag.prune.daemon.run.interval = 5m
> ** ozone.scm.block.size = 1KB
>
> OM Error logs -
> {{}}
> {code:java}
> 2025-06-03 05:22:20,972 INFO
> [om1546335944-impl-thread1]-org.apache.ratis.server.storage.RaftStorageDirectory:
> Lock on
> /var/lib/hadoop-ozone/om/ratis/2b34c18d-ed7a-3124-99bd-ca13a35e4b88/in_use.lock
> acquired by nodename [email protected]
> 2025-06-03 05:22:20,978 ERROR
> [main]-org.apache.hadoop.ozone.om.OzoneManagerStarter: OM start failed with
> exception
> java.util.concurrent.CompletionException: java.io.IOException: Failed to load
> /var/lib/hadoop-ozone/om/ratis/2b34c18d-ed7a-3124-99bd-ca13a35e4b88/current/raft-meta
> at
> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
> at
> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
> at
> java.base/java.util.concurrent.CompletableFuture$BiRelay.tryFire(CompletableFuture.java:1498)
> at
> java.base/java.util.concurrent.CompletableFuture$CoCompletion.tryFire(CompletableFuture.java:1219)
> at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> at
> java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
> at
> org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:191)
> at
> org.apache.ratis.util.ConcurrentUtils.lambda$null$4(ConcurrentUtils.java:180)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> at java.base/java.lang.Thread.run(Thread.java:833)
> Caused by: java.io.IOException: Failed to load
> /var/lib/hadoop-ozone/om/ratis/2b34c18d-ed7a-3124-99bd-ca13a35e4b88/current/raft-meta
> at
> org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.load(RaftStorageMetadataFileImpl.java:118)
> at
> org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.lambda$getMetadata$0(RaftStorageMetadataFileImpl.java:55)
> at
> org.apache.ratis.util.ConcurrentUtils.lambda$updateAndGet$0(ConcurrentUtils.java:51)
> at
> java.base/java.util.concurrent.atomic.AtomicReference.updateAndGet(AtomicReference.java:210)
> at
> org.apache.ratis.util.ConcurrentUtils.updateAndGet(ConcurrentUtils.java:49)
> at
> org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getMetadata(RaftStorageMetadataFileImpl.java:55)
> at
> org.apache.ratis.server.storage.RaftStorageImpl.analyzeAndRecoverStorage(RaftStorageImpl.java:116)
> at
> org.apache.ratis.server.storage.RaftStorageImpl.initialize(RaftStorageImpl.java:63)
> at
> org.apache.ratis.server.impl.ServerState.initialize(ServerState.java:169)
> at
> org.apache.ratis.server.impl.RaftServerImpl.start(RaftServerImpl.java:338)
> at
> org.apache.ratis.util.ConcurrentUtils.accept(ConcurrentUtils.java:188)
> ... 4 more
> Caused by: java.io.IOException: Failed to parse 'term' from properties: {}
> at
> org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getTerm(RaftStorageMetadataFileImpl.java:96)
> at
> org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.load(RaftStorageMetadataFileImpl.java:116)
> ... 14 more
> Caused by: java.io.IOException: 'term' not found in properties: {}
> at
> org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.lambda$getValue$2(RaftStorageMetadataFileImpl.java:89)
> at java.base/java.util.Optional.orElseThrow(Optional.java:403)
> at
> org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getValue(RaftStorageMetadataFileImpl.java:88)
> at
> org.apache.ratis.server.storage.RaftStorageMetadataFileImpl.getTerm(RaftStorageMetadataFileImpl.java:94)
> ... 15 more
> 2025-06-03 05:22:20,982 INFO
> [shutdown-hook-0]-org.apache.ranger.audit.provider.AuditProviderFactory: ==>
> JVMShutdownHook.run() {code}
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
