DevMattG opened a new pull request, #11048: URL: https://github.com/apache/arrow-rs/pull/11048
# Which issue does this PR close? - Closes #3478. # Rationale for this change `FlightDataEncoder` splits record batches based on a cheap, buffer-memory-size heuristic before IPC encoding. That heuristic can be significantly wrong -- it ignores IPC framing/padding overhead, and undercounts batches with dictionaries or highly skewed row sizes -- so pieces produced by the initial split can still end up well over `max_flight_data_size` once actually encoded. # What changes are included in this PR? - After the initial heuristic split, each piece is now IPC-encoded and its *actual* encoded size is checked against `max_flight_data_size` in `FlightDataEncoder::encode_piece_adaptive`. If it's still too large, the piece is re-split using the real size ratio and the sub-pieces are recursively re-checked, so oversized pieces get corrected regardless of why the initial estimate was wrong. - Dictionaries are queued unconditionally as soon as they're encoded, independent of whether the record batch carrying them ends up split further. - Fixed `split_batch_for_grpc_response`'s batch-count calculation to round up (`div_ceil`) instead of using floor division, which could previously produce one too few batches. # Are these changes tested? Yes. Added/extended unit tests in `arrow-flight/src/encode.rs` covering: - Repeated/deep adaptive re-splitting under a heavily skewed dictionary size distribution. - Adaptive re-splitting with both `DictionaryHandling::Resend` and the default `DictionaryHandling::Hydrate`. - A single row whose own encoded size exceeds `max_flight_data_size` (can't be split further, must still be emitted). # Are there any user-facing changes? No breaking API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
