Hmm... it looks like this was missed in our triage. I'm looking into why. I'm glad you filed https://github.com/grpc/grpc-java/issues/6685 for the okhttp bug. Ideally you would use Netty, but it is strange that Netty is slower than OkHttp. My best guess (based on zero evidence) is something is going awry with Zulu. Dunno though.
What you see would be classic behavior for "there's a bug in our flow control handling." Although it is a bit weird you hit it so quickly. I'd expect it to be a more subtle bug. We can continue the conversation on the issue you opened, and on a "netty is slower than okhttp" issue. It seems like there's a problem with the code; it seems unlikely "you're holding it wrong." On Fri, Jan 31, 2020 at 2:25 PM Christopher Schechter <[email protected]> wrote: > Hello, > > Running into a mysterious issue, and would really appreciate some insight. > > We're using gRPC server-side streaming to transfer some data between two > services (via protocol buffers). We're working on performance improvements > as the system isn't as fast as it needs to be. > > We set up a test environment as follows: > > - a client that initiates the server-side stream and immediately > discards messages as they're received > - a server that produces a stream of uniformly-sized messages > consisting of a random ByteString wrapped in a protocol buffer, sending > them as fast as possible > - both client and server run in separate processes on the same machine > > We tried switching the client from Netty channels to Okhttp channels. > Testing with the above setup showed a 4x speed increase, which is awesome. > However, then we introduced a proxy between the client & server that adds > 100ms of latency (simulating a connection that goes between our production > datacenters). The rate while using Okhttp dropped from 200,000 messages/sec > to 2,000 messages/sec. > > We theorized this could be solved by increasing the clients' > flowControlWindow (especially since the Okhttp default is much smaller than > Netty's). However, increasing the window size by any significant amount > (more than a few kb) results in the client receiving a few hundred messages > and then stopping. According to the logs, the stream remains open until the > client terminates it due to deadline, then the client establishes a new > stream on the same connection, but still no more messages are sent. We > can't find a reason for this to happen. (This doesn't happen with netty) > > > -- > You received this message because you are subscribed to the Google Groups " > grpc.io" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/grpc-io/21beb5c7-cffe-40bc-9d46-a67298871cf6%40googlegroups.com > <https://groups.google.com/d/msgid/grpc-io/21beb5c7-cffe-40bc-9d46-a67298871cf6%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CA%2B4M1oM6a3G_nfjnLBA2EpzMC5qhds1%2BGxY51XOh5vFKrEHAyg%40mail.gmail.com.
smime.p7s
Description: S/MIME Cryptographic Signature
