lidavidm opened a new pull request #7299: URL: https://github.com/apache/arrow/pull/7299
I'd appreciate feedback on this. This is a specialized API to limit the payload size when a client is sending data to a server. While Flight by default removes the gRPC-level limits on message sizes, we've found setting server-side limits on the size of incoming messages is still useful to ensure clients don't overwhelm a server/exhaust server memory by sending very large messages. However, if you exceed the limit at the gRPC level, the server resets the connection, making it hard for a client to recover gracefully. This PR adds an optional client-side check, so the client can catch the error and retry with a smaller record batch, or provide a meaningful error to the user if the batch cannot be made small enough. Unlike solutions involving `pa.ipc.get_record_batch_size`, this avoids duplicate re-serialization of the record batch. This is essentially a flight-specific version of [ARROW-5377](https://issues.apache.org/jira/browse/ARROW-5377). I proposed a candidate API there, but it's much more complex than the solution here. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
