[
https://issues.apache.org/jira/browse/HDDS-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-1414:
-------------------------------
Description:
TestOzoneManagerHA.testMultipartUploadWithOneOmNodeDown is flaky, we get the
below exception when it fails.
{code}
org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client
client-04649B8D5AF3->RAFT is closed.
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommand(OzoneManagerRatisClient.java:133)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestToRatis(OzoneManagerProtocolServerSideTranslatorPB.java:97)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:83)
at
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
Caused by: org.apache.ratis.protocol.AlreadyClosedException:
SlidingWindow$Client client-04649B8D5AF3->RAFT is closed.
at
org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:350)
at
org.apache.ratis.util.SlidingWindow$Client.submitNewRequest(SlidingWindow.java:224)
at
org.apache.ratis.client.impl.RaftClientImpl.sendAsync(RaftClientImpl.java:207)
at
org.apache.ratis.client.impl.RaftClientImpl.sendAsync(RaftClientImpl.java:174)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendRequestAsync(OzoneManagerRatisClient.java:208)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommandAsync(OzoneManagerRatisClient.java:168)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommand(OzoneManagerRatisClient.java:132)
... 11 more
Caused by: org.apache.ratis.protocol.RaftRetryFailureException: Failed
RaftClientRequest:client-04649B8D5AF3->omNode-1@group-523986131536, cid=71,
seq=1*, RW,
org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$396/1529424209@7ae5da75
for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms)
at
org.apache.ratis.client.impl.RaftClientImpl.newRaftRetryFailureException(RaftClientImpl.java:383)
at
org.apache.ratis.client.impl.RaftClientImpl.handleAsyncRetryFailure(RaftClientImpl.java:388)
at
org.apache.ratis.client.impl.RaftClientImpl.lambda$sendRequestAsync$14(RaftClientImpl.java:370)
at
java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
at
java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:329)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:245)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:257)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:248)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:421)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:519)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
> TestOzoneManagerHA.testMultipartUploadWithOneOmNodeDown is flaky
> ----------------------------------------------------------------
>
> Key: HDDS-1414
> URL: https://issues.apache.org/jira/browse/HDDS-1414
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Elek, Marton
> Priority: Major
> Labels: ozone-flaky-test
> Attachments: ci2.log
>
>
> TestOzoneManagerHA.testMultipartUploadWithOneOmNodeDown is flaky, we get the
> below exception when it fails.
> {code}
> org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client
> client-04649B8D5AF3->RAFT is closed.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommand(OzoneManagerRatisClient.java:133)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestToRatis(OzoneManagerProtocolServerSideTranslatorPB.java:97)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:83)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> Caused by: org.apache.ratis.protocol.AlreadyClosedException:
> SlidingWindow$Client client-04649B8D5AF3->RAFT is closed.
> at
> org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:350)
> at
> org.apache.ratis.util.SlidingWindow$Client.submitNewRequest(SlidingWindow.java:224)
> at
> org.apache.ratis.client.impl.RaftClientImpl.sendAsync(RaftClientImpl.java:207)
> at
> org.apache.ratis.client.impl.RaftClientImpl.sendAsync(RaftClientImpl.java:174)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendRequestAsync(OzoneManagerRatisClient.java:208)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommandAsync(OzoneManagerRatisClient.java:168)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient.sendCommand(OzoneManagerRatisClient.java:132)
> ... 11 more
> Caused by: org.apache.ratis.protocol.RaftRetryFailureException: Failed
> RaftClientRequest:client-04649B8D5AF3->omNode-1@group-523986131536, cid=71,
> seq=1*, RW,
> org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisClient$$Lambda$396/1529424209@7ae5da75
> for 10 attempts with RetryLimited(maxAttempts=10, sleepTime=100ms)
> at
> org.apache.ratis.client.impl.RaftClientImpl.newRaftRetryFailureException(RaftClientImpl.java:383)
> at
> org.apache.ratis.client.impl.RaftClientImpl.handleAsyncRetryFailure(RaftClientImpl.java:388)
> at
> org.apache.ratis.client.impl.RaftClientImpl.lambda$sendRequestAsync$14(RaftClientImpl.java:370)
> at
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
> at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:329)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:245)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:257)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:248)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:421)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:519)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
