alamb commented on issue #7812: URL: https://github.com/apache/arrow-rs/issues/7812#issuecomment-4856626050
This came up in Datafusion: https://github.com/apache/datafusion/issues/23247 I would actually like to propose a slightly different API here: a "sans IO" encoder -- specfically an encoder that does no IO I recommend something like the BatchCoalecer API: https://docs.rs/arrow/latest/arrow/compute/struct.BatchCoalescer.html where the output is written to buffers ```rust // Not there is no IO / stream: let mut encoder = StreamEncoder::try_new_with_options(&schema, options).unwrap(); // Encode a batch to buffers writer.encode(&batch1).unwrap(); // for buffer in writer.take_buffers() { // bufers is Vec<Vec<u8>> that has encoded IPC data and can be // written to a stream, etc } ``` I think this could be a pretty thin wrapper around https://docs.rs/arrow-ipc/latest/arrow_ipc/writer/struct.IpcDataGenerator.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
