The 40M that Carl mentions makes my eyes water. You're going to have to tweak your network *so* hard to make that work.
In a case like that, I would strongly recommend doing what Robert Engels suggested -- use the streaming protocol and send it in chunks. If you need transactional atomicity, you just send a "OK, that's the end of the batch, commit this now" message as part of the stream, and essentially have your server accumulate the data until it's ready to go. This is going to be *way* more robust and reliable, and not require you to do anything too unspeakable to networks or the like. (I don't remember if the gRPC API exposes raw messages, but if it does and you are for some reason receiving a 40M protobuf that you want to chunk up, the wire format of a serialized protobuf lends itself very naturally to splitting, either by replacing SerializeToString() with your own logic, or by chopping up the output. OTOH, if you're receiving a single 40M protobuf in a single lump and that isn't because one field is 40MB of raw bytes, you might want to rethink your data structures on that end too, to make managing it easier. I've had to work with cases where these things scaled much bigger -- a distributed storage system where multi-T files were not uncommon things to deal with -- and setting a chunk size of ~256kB was a very important thing for all sorts of reasons, not just for making gRPC happy) Yonatan On Thursday, December 13, 2018 at 10:14:06 AM UTC-8, Oregonduckman wrote: > > I have read several posts about the maximum size gRPC will transport and > posts range from 4M to 2G. I am interested to know what others are doing to > transport large buffers on the order of 200M using gRPC. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/a6a19210-5223-4dc8-8fd9-39078bd70ba9%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
