jkammerer opened a new issue, #37900: URL: https://github.com/apache/arrow/issues/37900
### Describe the enhancement requested We use Arrow flight over GRPC in C++ to request large volumes of data from another service. This data is streamed to us using potentially large individual messages. Each request uses its own `grpc::channel`. To guarantee the stability of our service, we need to be able to track the memory that is allocated for a request. When reading a message, we see that the process memory intermittently increases by the message’s size. Since we have messages of multiple megabytes, we need to be able to track these internal allocations. It is important for us that this memory tracking happens before the allocation, such that, we can decide to either allow or forbid the allocation based on our memory budget. Our problems could be solved with some kind of custom allocator support or if we could get the size of the next incoming message before it is completely read into memory. Using GRPC’s [`NextMessageSize()`](https://grpc.github.io/grpc/cpp/classgrpc_1_1internal_1_1_reader_interface.html#ad3216783fae57e614181e28df9435f33) does not fit our needs as it always returns the set `MaxReceiveMessageSize` of the underlying GRPC channel. However, we would need to know the actual upcoming message size because our maximum message size is a lot larger than what we expect to receive in the “normal” cases. Always expecting the maximum message size would lead to large amounts of unused memory being held in reserve by our system. We also investigated whether this problem can be solved on the GRPC level but did not find a satisfying solution. We opened [an issue for this](https://github.com/grpc/grpc/issues/34498) as well. ### Component(s) C++, FlightRPC -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
