[
https://issues.apache.org/jira/browse/RATIS-977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
runzhiwang updated RATIS-977:
-----------------------------
Description:
1. s2 reply to s3, but s3 failed to read message.
{code:java}
2020-06-14T13:20:21.5332706Z 2020-06-14 13:20:21,532 [Thread-193] DEBUG
impl.RaftServerImpl (RaftServerImpl.java:logAppendEntries(916)) -
s2@group-A42C745E6D8C: succeeded to handle AppendEntries. Reply:
s3<-s2#78:OK,SUCCESS,nextIndex:3,term:2,followerCommit:1,matchIndex:2
2020-06-14T13:20:21.5483383Z 2020-06-14 13:20:21,545 [grpc-default-executor-4]
WARN server.GrpcLogAppender (LogUtils.java:warn(122)) -
s3@group-A42C745E6D8C->s2-AppendLogResponseHandler: Failed appendEntries:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: Failed
to read message.
{code}
2. The stack is as follow:
{code:java}
2020-06-15T02:09:05.6092588Z
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: Failed
to read message.
2020-06-15T02:09:05.6092751Z at
org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:533)
2020-06-15T02:09:05.6092920Z at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:449)
2020-06-15T02:09:05.6093083Z at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
2020-06-15T02:09:05.6093245Z at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
2020-06-15T02:09:05.6093420Z at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
2020-06-15T02:09:05.6093787Z at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
2020-06-15T02:09:05.6093967Z at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:670)
2020-06-15T02:09:05.6094134Z at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:643)
2020-06-15T02:09:05.6094300Z at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2020-06-15T02:09:05.6094460Z at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
2020-06-15T02:09:05.6094615Z at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2020-06-15T02:09:05.6094846Z at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2020-06-15T02:09:05.6094997Z at java.lang.Thread.run(Thread.java:748)
2020-06-15T02:09:05.6095159Z Caused by:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: Invalid
protobuf byte sequence
2020-06-15T02:09:05.6095316Z at
org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:524)
2020-06-15T02:09:05.6095480Z at
org.apache.ratis.thirdparty.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:218)
2020-06-15T02:09:05.6095649Z at
org.apache.ratis.thirdparty.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:118)
2020-06-15T02:09:05.6095816Z at
org.apache.ratis.thirdparty.io.grpc.MethodDescriptor.parseResponse(MethodDescriptor.java:275)
2020-06-15T02:09:05.6095972Z at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:658)
2020-06-15T02:09:05.6096126Z ... 6 more
2020-06-15T02:09:05.6096283Z Caused by:
org.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
Protocol message contained an invalid tag (zero).
2020-06-15T02:09:05.6096473Z at
org.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:102)
2020-06-15T02:09:05.6096697Z at
org.apache.ratis.thirdparty.com.google.protobuf.CodedInputStream$ArrayDecoder.readTag(CodedInputStream.java:627)
2020-06-15T02:09:05.6096864Z at
org.apache.ratis.proto.RaftProtos$AppendEntriesReplyProto.<init>(RaftProtos.java:16335)
{code}
was:
1. s2 reply to s3, but s3 failed to read message.
{code:java}
2020-06-14T13:20:21.5332706Z 2020-06-14 13:20:21,532 [Thread-193] DEBUG
impl.RaftServerImpl (RaftServerImpl.java:logAppendEntries(916)) -
s2@group-A42C745E6D8C: succeeded to handle AppendEntries. Reply:
s3<-s2#78:OK,SUCCESS,nextIndex:3,term:2,followerCommit:1,matchIndex:2
2020-06-14T13:20:21.5483383Z 2020-06-14 13:20:21,545 [grpc-default-executor-4]
WARN server.GrpcLogAppender (LogUtils.java:warn(122)) -
s3@group-A42C745E6D8C->s2-AppendLogResponseHandler: Failed appendEntries:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: Failed
to read message.
{code}
2. The stack is as follow:
{code:java}
2020-06-14T13:20:21.5454979Z
org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler|GrpcLogAppender.java|319|onError
2020-06-14T13:20:21.5456799Z
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter|ClientCalls.java|449|onClose
2020-06-14T13:20:21.5458327Z
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl|ClientCallImpl.java|426|closeObserver
2020-06-14T13:20:21.5459644Z
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl|ClientCallImpl.java|66|access$500
2020-06-14T13:20:21.5461371Z
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl|ClientCallImpl.java|689|close
2020-06-14T13:20:21.5463161Z
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl|ClientCallImpl.java|577|access$900
2020-06-14T13:20:21.5465434Z
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable|ClientCallImpl.java|670|runInternal
2020-06-14T13:20:21.5467123Z
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable|ClientCallImpl.java|643|runInContext
2020-06-14T13:20:21.5468555Z
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable|ContextRunnable.java|37|run
2020-06-14T13:20:21.5470018Z
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor|SerializingExecutor.java|123|run
2020-06-14T13:20:21.5471187Z
java.util.concurrent.ThreadPoolExecutor|ThreadPoolExecutor.java|1149|runWorker
2020-06-14T13:20:21.5472376Z
java.util.concurrent.ThreadPoolExecutor$Worker|ThreadPoolExecutor.java|624|run
2020-06-14T13:20:21.5473033Z java.lang.Thread|Thread.java|748|run
{code}
> Fix gRPC failed to read message
> -------------------------------
>
> Key: RATIS-977
> URL: https://issues.apache.org/jira/browse/RATIS-977
> Project: Ratis
> Issue Type: Bug
> Reporter: runzhiwang
> Assignee: runzhiwang
> Priority: Major
>
> 1. s2 reply to s3, but s3 failed to read message.
> {code:java}
> 2020-06-14T13:20:21.5332706Z 2020-06-14 13:20:21,532 [Thread-193] DEBUG
> impl.RaftServerImpl (RaftServerImpl.java:logAppendEntries(916)) -
> s2@group-A42C745E6D8C: succeeded to handle AppendEntries. Reply:
> s3<-s2#78:OK,SUCCESS,nextIndex:3,term:2,followerCommit:1,matchIndex:2
> 2020-06-14T13:20:21.5483383Z 2020-06-14 13:20:21,545
> [grpc-default-executor-4] WARN server.GrpcLogAppender
> (LogUtils.java:warn(122)) -
> s3@group-A42C745E6D8C->s2-AppendLogResponseHandler: Failed appendEntries:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: Failed
> to read message.
> {code}
> 2. The stack is as follow:
> {code:java}
> 2020-06-15T02:09:05.6092588Z
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: Failed
> to read message.
> 2020-06-15T02:09:05.6092751Z at
> org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:533)
> 2020-06-15T02:09:05.6092920Z at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:449)
> 2020-06-15T02:09:05.6093083Z at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
> 2020-06-15T02:09:05.6093245Z at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
> 2020-06-15T02:09:05.6093420Z at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
> 2020-06-15T02:09:05.6093787Z at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
> 2020-06-15T02:09:05.6093967Z at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:670)
> 2020-06-15T02:09:05.6094134Z at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:643)
> 2020-06-15T02:09:05.6094300Z at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> 2020-06-15T02:09:05.6094460Z at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> 2020-06-15T02:09:05.6094615Z at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-06-15T02:09:05.6094846Z at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-06-15T02:09:05.6094997Z at java.lang.Thread.run(Thread.java:748)
> 2020-06-15T02:09:05.6095159Z Caused by:
> org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL: Invalid
> protobuf byte sequence
> 2020-06-15T02:09:05.6095316Z at
> org.apache.ratis.thirdparty.io.grpc.Status.asRuntimeException(Status.java:524)
> 2020-06-15T02:09:05.6095480Z at
> org.apache.ratis.thirdparty.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:218)
> 2020-06-15T02:09:05.6095649Z at
> org.apache.ratis.thirdparty.io.grpc.protobuf.lite.ProtoLiteUtils$MessageMarshaller.parse(ProtoLiteUtils.java:118)
> 2020-06-15T02:09:05.6095816Z at
> org.apache.ratis.thirdparty.io.grpc.MethodDescriptor.parseResponse(MethodDescriptor.java:275)
> 2020-06-15T02:09:05.6095972Z at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:658)
> 2020-06-15T02:09:05.6096126Z ... 6 more
> 2020-06-15T02:09:05.6096283Z Caused by:
> org.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
> Protocol message contained an invalid tag (zero).
> 2020-06-15T02:09:05.6096473Z at
> org.apache.ratis.thirdparty.com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:102)
> 2020-06-15T02:09:05.6096697Z at
> org.apache.ratis.thirdparty.com.google.protobuf.CodedInputStream$ArrayDecoder.readTag(CodedInputStream.java:627)
> 2020-06-15T02:09:05.6096864Z at
> org.apache.ratis.proto.RaftProtos$AppendEntriesReplyProto.<init>(RaftProtos.java:16335)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
