Rich-T-kid commented on PR #11048: URL: https://github.com/apache/arrow-rs/pull/11048#issuecomment-6837385171
>@DevMattG @Rich-T-kid I believe both this PR and https://github.com/apache/arrow-rs/pull/10383 solve https://github.com/apache/arrow-rs/issues/3478 from different directions. This PR makes max_flight_data_size a hard limit (documentation should be updated in this PR). 👍 I agree >But there is still an issue where the memory size is overestimated because the size of the underlying buffer is taken instead of the size of the encoded slice. The notable pathological case is when you want to reencode the batch that was produced during the IPC serialization. Each column counts the whole message as its own buffer size, so it depends on the schema, but often the end result will be the split which split the batch into single row batches. I think it makes sense to incorporate https://github.com/apache/arrow-rs/pull/10383 into this PR, what do you guys think? Thank you both for working on https://github.com/apache/arrow-rs/issues/3478, hopefully this can be resolved and merged soon. I agree. would be nice to solve both of these. typically Id be in favor of splitting this into two PR's but I think they are quite related -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
