Elek, Marton created HDDS-1414:
----------------------------------
Summary: TestOzoneManagerHA.testMultipartUploadWithOneOmNodeDown
is flaky
Key: HDDS-1414
URL: https://issues.apache.org/jira/browse/HDDS-1414
Project: Hadoop Distributed Data Store
Issue Type: Bug
Environment: TestOzoneManagerHA.testMultipartUploadWithOneOmNodeDown
is flaky, we get the below exception when it fails.
{code}
org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client
client-04649B8D5AF3->RAFT is closed.
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommand(OzoneManagerRatisClient.java:133)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestToRatis(OzoneManagerProtocolServerSideTranslatorPB.java:97)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:83)
at
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
Caused by: org.apache.ratis.protocol.AlreadyClosedException:
SlidingWindow$Client client-04649B8D5AF3->RAFT is closed.
at
org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:350)
at
org.apache.ratis.util.SlidingWindow$Client.submitNewRequest(SlidingWindow.java:224)
at
org.apache.ratis.client.impl.RaftClientImpl.sendAsync(RaftClientImpl.java:207)
at
org.apache.ratis.client.impl.RaftClientImpl.sendAsync(RaftClientImpl.java:174)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendRequestAsync(OzoneManagerRatisClient.java:208)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommandAsync(OzoneManagerRatisClient.java:168)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommand(OzoneManagerRatisClient.java:132)
... 11 more
Caused by: org.apache.ratis.protocol.RaftRetryFailureException: Failed
RaftClientRequest:client-04649B8D5AF3->omNode-1@group-523986131536, cid=71,
seq=1*, RW,
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$396/1529424209@7ae5da75
for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms)
at
org.apache.ratis.client.impl.RaftClientImpl.newRaftRetryFailureException(RaftClientImpl.java:383)
at
org.apache.ratis.client.impl.RaftClientImpl.handleAsyncRetryFailure(RaftClientImpl.java:388)
at
org.apache.ratis.client.impl.RaftClientImpl.lambda$sendRequestAsync$14(RaftClientImpl.java:370)
at
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
at
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:329)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:245)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:257)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:248)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:421)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:519)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
Reporter: Elek, Marton
Attachments: ci2.log
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
