Jyotirmoy Sinha created HDDS-12479:
--------------------------------------
Summary: Datanode down due to RejectedExecutionException
Key: HDDS-12479
URL: https://issues.apache.org/jira/browse/HDDS-12479
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Datanode
Reporter: Jyotirmoy Sinha
Scenario :
* Pump data into the cluster to fill up cluster for more 90%. Data consists of
hive, tera, freon and ozone keys.
* Datanode down due to RejectedExecutionException
Datanode error stacktrace -
{code:java}
2025-02-03 05:18:51,108 INFO
[grpc-default-executor-12]-org.apache.ratis.grpc.server.GrpcServerProtocolService:
5b2cbbd1-6fe2-4304-9618-a7479328f7a3: Completed APPEND_ENTRIES, lastRequest:
023b20ba-d7e5-4492-875b-94779796a544->5b2cbbd1-6fe2-4304-9618-a7479328f7a3#1109-t4,previous=(t:4,
i:1030),leaderCommit=1030,initializing? false,entries: size=2, first=(t:4,
i:1031), METADATAENTRY(c:1029)
2025-02-03 05:18:51,108 INFO
[grpc-default-executor-12]-org.apache.ratis.grpc.server.GrpcServerProtocolService:
5b2cbbd1-6fe2-4304-9618-a7479328f7a3: Completed APPEND_ENTRIES, lastReply: null
2025-02-03 05:18:51,269 WARN
[grpc-default-executor-12]-org.apache.ratis.grpc.server.GrpcLogAppender:
5b2cbbd1-6fe2-4304-9618-a7479328f7a3@group-C08DE0CB48EA->023b20ba-d7e5-4492-875b-94779796a544-AppendLogResponseHandler:
Failed appendEntries
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.CompletableFuture$AsyncSupply@235fd375 rejected from
java.util.concurrent.ThreadPoolExecutor@7cd08047[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 1054]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1618)
at
java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:1843)
at
org.apache.ratis.server.impl.RaftServerImpl.executeSubmitServerRequestAsync(RaftServerImpl.java:886)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$appendEntriesAsync$29(RaftServerProxy.java:652)
at
java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:995)
at
java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2137)
at
org.apache.ratis.server.impl.RaftServerProxy.appendEntriesAsync(RaftServerProxy.java:652)
at
org.apache.ratis.grpc.server.GrpcServerProtocolService$1.process(GrpcServerProtocolService.java:229)
at
org.apache.ratis.grpc.server.GrpcServerProtocolService$1.process(GrpcServerProtocolService.java:226)
at
org.apache.ratis.grpc.server.GrpcServerProtocolService$ServerRequestStreamObserver.onNext(GrpcServerProtocolService.java:137)
at
org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:329)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:314)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:833)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748){code}
{code:java}
2025-02-11 19:27:45,290 INFO
[5b2cbbd1-6fe2-4304-9618-a7479328f7a3-BlockDeletingService#5]-org.apache.hadoop.ozone.container.common.interfaces.ContainerDeletionChoosingPolicyTemplate:
Chosen 0/5000 blocks from 0 candidate containers.
2025-02-11 19:27:53,291 WARN
[grpc-default-worker-ELG-3-18]-org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor:
A task raised an exception. Task:
org.apache.ratis.thirdparty.io.grpc.netty.NettyServerStream$TransportState$1@777bcb22
java.util.concurrent.RejectedExecutionException: Task
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$ContextCloser@1db5008
rejected from java.util.concurrent.ThreadPoolExecutor@c9810af[Shutting down,
pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 180991]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener.closedInternal(ServerImpl.java:895)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener.closed(ServerImpl.java:876)
at
org.apache.ratis.thirdparty.io.grpc.internal.AbstractServerStream$TransportState.closeListener(AbstractServerStream.java:339)
at
org.apache.ratis.thirdparty.io.grpc.internal.AbstractServerStream$TransportState.access$100(AbstractServerStream.java:184)
at
org.apache.ratis.thirdparty.io.grpc.internal.AbstractServerStream$TransportState$1.run(AbstractServerStream.java:289)
at
org.apache.ratis.thirdparty.io.grpc.internal.AbstractServerStream$TransportState.deframerClosed(AbstractServerStream.java:241){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
