Hey Carl, Thank you for the response. Going to check this out later today.
Hollin On Wed, Feb 8, 2017 at 09:00 Carl Mastrangelo <[email protected]> wrote: > gRPC does have a backpressure mechanism via HTTP/2 flow control. I am not > as familiar with the Go mechanism for this, but at least in java it is > possible to check if the StreamObserver can proceed to send. As the server > receives data, it will send back more flow control window (as WINDOW_UPDATE > frames) for the client to proceed. > > To do this, cast the observer to a CallStreamObserver, and check isReady() > if you can send without buffering. You can also call setOnReadyHandler if > you don't want to poll. > > > On Tuesday, February 7, 2017 at 10:01:01 AM UTC-8, [email protected] > wrote: > > Hey everyone, > > I have a problem that I am unsure what to do about. I am creating a client > for a server. The server takes in a stream of file chunks and when the > client stream completes, the server sends back a single Empty response. The > Protobuf definition looks like this: > > rpc PutFile(stream PutFileRequest) returns (google.protobuf.Empty) {} > > Each PutFileRequest contains a chunk of the file. See the full definition > here: > https://github.com/pachyderm/pachyderm/blob/master/src/client/pfs/pfs.proto > > My issue is this. If I have a very large file, say 100gb and I only have > 4gb of memory and I want to send my file across the network, I have no way > to backpressure the system to avoid out of memory issues. Here is why: > > We use an instance of StreamObserver<PutFileRequest> from the client to > send chunks of the file. I have a process that is reading chunks of the > file from disk as fast as it can and sends them over the > StreamObserver<PutFileRequest> object. I have no way to know when the > request has been sent over the network and the memory for a chunk has been > released. Because of this, I will easily encounter an out of memory error > as I am reading from the disk at a much faster rate than I can send over > the network. > > Is there a way to be notified from gRPC when a specific message in a > stream has completed sending? I could listen for this event and slow down > the rate at which I am reading from disk accordingly. > > Thank you, > Hollin Wilkins > > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAJU54TDOJjYTX05XvroroivCgEE%3DvvR80wg6Umg_73ztOuiV8A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
