matthewmturner commented on issue #208: URL: https://github.com/apache/arrow-rs/issues/208#issuecomment-917692820
Thanks much for the response @alamb > I am not sure to be honest as I am not familiar with the flight code. Perhaps @nevi-me or @jorgecarleitao who have more experience in how IPC / flight is supposed to work might have thoughts on how to handle serializing bytes for an Array whose backing `Buffer` is much larger. Another avenue we can explore is to review how the C++ implementation handles the case and/or ask about this on [[email protected]](mailto:[email protected]). > > One way to reduce potential unintended side effects could be to make the optimization optional (an option on [`IpcWriteOptions`](https://docs.rs/arrow/5.3.0/arrow/ipc/writer/struct.IpcWriteOptions.html), perhaps) while we test it out more broadly, and then switch the default value in a later version. > I checked the `IpcWriteOptions` for the C++ implementation (http://arrow.apache.org/docs/cpp/api/ipc.html) and I dont think they have an option for that. It also doesnt look like there is a user option for this in their Flight client implementation (http://arrow.apache.org/docs/cpp/api/flight.html#_CPPv4N5arrow6flight17FlightCallOptionsE). I can email the dev email list to see if they have any hidden logic for this not exposed to the end user. > `RecordBatch::slice` is what I know of for this purpose: https://docs.rs/arrow/5.3.0/arrow/record_batch/struct.RecordBatch.html#method.slice. (Kudos to @b41sh for adding that one) > Thanks will check that out! Not sure how i missed that function. @nevi-me @jorgecarleitao any thoughts or preferences on how to handle this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
