For reference, https://github.com/grpc/grpc/issues/27292 is the related
issue
On Thursday, August 26, 2021 at 10:09:41 AM UTC-7 Jacob B wrote:
> We are using gRPC (version 1.37.1) for our inter-process communication
> between our C# process and C++ process. Both processes act as a server and
> client with the other and run on the same machine over localhost using the
> HTTP/2 transport. All of the calls are use blocking synchronous unary calls
> and not bi-directional streaming. Some average(ish) stats:
>
> From C++->C#: 0-2 calls per second, 0-40 calls per minute
>
> From C#->C++: 0-5 calls per second, 0-200 calls per minute
>
> Intermittently, we were getting one of 3 issues
>
> - C# client call to C++ server comes back with an RpcException,
> usually “HTTP2/Parse Error”, “Endpoint Read Failed”, or “Transport Closed”
> - C++ client call to C# server comes back with Unavailable or Unknown
> - C++ client WaitForConnected call to check the channel fails after
> 500ms
>
>
>
> The top most one is the most frequent and where we have the most
> information about. Usually, what we’ll see is the Client receives the RPC
> call and runs into an unknown frame type. Then the subchannel goes into
> shutdown and everything usually re-connects fine. We also generally see an
> embedded error like the following (note that we replaced all __FILE__
> instances to __FUNCTION__ in our gRPC source):
>
> win_read","file_line":307,"os_error":"The system detected an invalid
> pointer address in attempting to use a pointer argument in a
> call.\r\n","syscall":"WSARecv","wsa_error":10014}]},{"created":"@1622120588.494000000","description":"frame
>
> of size 262404 overflows local window of
> 65535","file":"grpc_core::chttp2::TransportFlowControl::ValidateRecvData","file_line":213}]}
>
> What we’ve seen with the unknown frame type, is that it parses the
> HEADERS, WINDOW_UPDATE, DATA, WINDOW_UPDATE and then gets a TCP: on_read
> without a corresponding READ and then tries to parse again. It’s this parse
> where it looks like the parser is at the wrong offset in the buffer,
> because it gets the unknown frame type, incoming frame size and incoming
> stream_id all map to the middle of the RPC call that it just parsed.
>
>
>
> The above was what we were encountering prior to a change to create a new
> channel for each rpc call. While we realize it is not great from a
> performance standpoint, we have seen increased stability since making the
> change. However, we still do occasionally get rpc exceptions. Now, the most
> common is “Unknown”/”Stream Removed” rather than the ones listed above.
>
>
> Any ideas on what might be going wrong is appreciated.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/grpc-io/5d029abb-c274-4a4d-a10c-0e806bb32bd4n%40googlegroups.com.
