Pratyush Bhatt created HDDS-8632:
------------------------------------
Summary: OrderedAsync: Failed to send request, Container in CLOSED
state
Key: HDDS-8632
URL: https://issues.apache.org/jira/browse/HDDS-8632
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Affects Versions: 1.4.0
Reporter: Pratyush Bhatt
Seeing this in HMaster logs:
{noformat}
2023-05-16 13:27:29,997 INFO org.apache.hadoop.hbase.master.HMaster:
Client=hrt_qa//10.64.62.40 disable IntegrationTestLazyCfLoading
2023-05-16 13:27:30,122 ERROR org.apache.ratis.client.impl.OrderedAsync: Failed
to send request, message=cmdType: WriteChunk
traceID: ""
containerID: 4
datanodeUuid: "34e01ebb-73d5-4358-a0e5-b0f60ab74036"
writeChunk {
blockID {
containerID: 4
localID: 111677748019200349
blockCommitSequenceId: 149723
replicaIndex: 0
}
chunkData {
chunkName: "111677748019200349_chunk_2"
offset: 83
len: 391
checksumData {
type: CRC32
bytesPerChecksum: 1048576
checksums: "\203\003N\362"
}
}
}
java.util.concurrent.CompletionException:
org.apache.ratis.protocol.exceptions.StateMachineException:
org.apache.hadoop.hdds.scm.container.common.helpers.ContainerNotOpenException
from Server 7477f3d7-d623-4d20-85b1-361d36570192@group-953FC5EB5D1C: Container
4 in CLOSED state
at
org.apache.ratis.client.impl.RaftClientImpl.handleRaftException(RaftClientImpl.java:322)
at
org.apache.ratis.client.impl.OrderedAsync.lambda$send$3(OrderedAsync.java:172)
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:98)
at
org.apache.ratis.client.impl.OrderedAsync$PendingOrderedRequest.setReply(OrderedAsync.java:59)
at
org.apache.ratis.util.SlidingWindow$RequestMap.setReply(SlidingWindow.java:144)
at
org.apache.ratis.util.SlidingWindow$Client.receiveReply(SlidingWindow.java:348)
at
org.apache.ratis.client.impl.OrderedAsync.lambda$sendRequest$9(OrderedAsync.java:248)
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.lambda$onNext$0(GrpcClientProtocolClient.java:322)
at java.util.Optional.ifPresent(Optional.java:159)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.handleReplyFuture(GrpcClientProtocolClient.java:378)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$100(GrpcClientProtocolClient.java:300)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:322)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:305)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:474)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.internal.DelayedClientCall$DelayedListener.onMessage(DelayedClientCall.java:452)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:646)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
