gRPC does have a backpressure mechanism via HTTP/2 flow control. I am not as familiar with the Go mechanism for this, but at least in java it is possible to check if the StreamObserver can proceed to send. As the server receives data, it will send back more flow control window (as WINDOW_UPDATE frames) for the client to proceed.
To do this, cast the observer to a CallStreamObserver, and check isReady() if you can send without buffering. You can also call setOnReadyHandler if you don't want to poll. On Tuesday, February 7, 2017 at 10:01:01 AM UTC-8, [email protected] wrote: > > Hey everyone, > > I have a problem that I am unsure what to do about. I am creating a client > for a server. The server takes in a stream of file chunks and when the > client stream completes, the server sends back a single Empty response. The > Protobuf definition looks like this: > > rpc PutFile(stream PutFileRequest) returns (google.protobuf.Empty) {} > > Each PutFileRequest contains a chunk of the file. See the full definition > here: > https://github.com/pachyderm/pachyderm/blob/master/src/client/pfs/pfs.proto > > My issue is this. If I have a very large file, say 100gb and I only have > 4gb of memory and I want to send my file across the network, I have no way > to backpressure the system to avoid out of memory issues. Here is why: > > We use an instance of StreamObserver<PutFileRequest> from the client to > send chunks of the file. I have a process that is reading chunks of the > file from disk as fast as it can and sends them over the > StreamObserver<PutFileRequest> object. I have no way to know when the > request has been sent over the network and the memory for a chunk has been > released. Because of this, I will easily encounter an out of memory error > as I am reading from the disk at a much faster rate than I can send over > the network. > > Is there a way to be notified from gRPC when a specific message in a > stream has completed sending? I could listen for this event and slow down > the rate at which I am reading from disk accordingly. > > Thank you, > Hollin Wilkins > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/07cf8b07-0a18-42e5-ae8a-efc5ca256521%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
