Jyotirmoy Sinha created HDDS-12480:
--------------------------------------
Summary: IllegalStateException in ozone datanode when cluster
capacity is more than 90%
Key: HDDS-12480
URL: https://issues.apache.org/jira/browse/HDDS-12480
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Reporter: Jyotirmoy Sinha
Scenario :
* Pump data into the cluster to fill up cluster for more 90%. Data consists of
hive, tera, freon and ozone keys.
* Multiple instances of IllegalStateException in the datanode
Datanode error stacktrace -
{code:java}
2025-02-11 20:59:34,019 WARN
[grpc-default-executor-4252]-org.apache.ratis.grpc.server.GrpcLogAppender:
4c4dabc6-00f0-4dba-a3a5-734727a22ad6@group-9466FC0E8B0F->5d1c4588-895b-4c11-b59c-31b66c70525d-AppendLogResponseHandler:
Failed appendEntries
java.io.IOException: java.lang.IllegalStateException:
5d1c4588-895b-4c11-b59c-31b66c70525d@group-9466FC0E8B0F-SegmentedRaftLog is
expected to be opened but it is CLOSED
at org.apache.ratis.util.IOUtils.asIOException(IOUtils.java:56)
at
org.apache.ratis.server.impl.RaftServerImpl.appendEntriesAsync(RaftServerImpl.java:1488)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$null$28(RaftServerProxy.java:652)
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:118)
at
org.apache.ratis.server.impl.RaftServerImpl.lambda$executeSubmitServerRequestAsync$10(RaftServerImpl.java:887)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException:
5d1c4588-895b-4c11-b59c-31b66c70525d@group-9466FC0E8B0F-SegmentedRaftLog is
expected to be opened but it is CLOSED
at
org.apache.ratis.util.OpenCloseState.assertOpen(OpenCloseState.java:63)
at
org.apache.ratis.server.raftlog.RaftLogBase.checkLogState(RaftLogBase.java:112)
at
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.getTermIndex(SegmentedRaftLog.java:341)
at org.apache.ratis.server.raftlog.RaftLog.contains(RaftLog.java:49)
at
org.apache.ratis.server.impl.ServerState.containsTermIndex(ServerState.java:501)
at
org.apache.ratis.server.impl.RaftServerImpl.checkInconsistentAppendEntries(RaftServerImpl.java:1649)
at
org.apache.ratis.server.impl.RaftServerImpl.appendEntriesAsync(RaftServerImpl.java:1579)
at
org.apache.ratis.server.impl.RaftServerImpl.appendEntriesAsync(RaftServerImpl.java:1484)
... 7 more
Caused by: org.apache.ratis.util.OpenCloseState$CloseTrace: Close
5d1c4588-895b-4c11-b59c-31b66c70525d@group-9466FC0E8B0F-SegmentedRaftLog
at
org.apache.ratis.util.OpenCloseState.lambda$close$1(OpenCloseState.java:109)
at
java.util.concurrent.atomic.AtomicReference.getAndUpdate(AtomicReference.java:160)
at org.apache.ratis.util.OpenCloseState.close(OpenCloseState.java:109)
at
org.apache.ratis.server.raftlog.RaftLogBase.close(RaftLogBase.java:391)
at
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog.close(SegmentedRaftLog.java:531)
at
org.apache.ratis.server.raftlog.RaftLogBase.lambda$appendImpl$3(RaftLogBase.java:205)
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLog$Task.failed(SegmentedRaftLog.java:111)
at
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker$WriteLog.failed(SegmentedRaftLogWorker.java:521)
at
org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker.run(SegmentedRaftLogWorker.java:319)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
