lidavidm commented on issue #13980: URL: https://github.com/apache/arrow/issues/13980#issuecomment-1228409993
I would try removing the backpressure wait and seeing how that helps: https://github.com/apache/arrow/blob/b832853ba62171d5fe5077681083fc6ea49bfd44/java/flight/flight-core/src/test/java/org/apache/arrow/flight/perf/PerformanceTestServer.java#L144 (this will require a lot of memory, you may have to manually add sleep() calls to space out the batches a bit) the issue with gRPC Java is that it has a fixed, small buffer it uses for backpressure, so effectively _every_ write will trigger backpressure and artificially throttle the producer, regardless of actual network conditions. This has been known for years but the upstream is not interested in fixing it: https://github.com/grpc/grpc-java/issues/5433 Unfortunately without it, Java _never_ applies any backpressure and instead you tend to just OOM. I believe C++ does the 'right' thing and automatically applies backpressure (by blocking the send call) above some threshold which I do not recall, but which I _think_ is actually based on network conditions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
