Phoenix500526 commented on issue #23247: URL: https://github.com/apache/datafusion/issues/23247#issuecomment-4851676608
> This is a good point [@pantShrey](https://github.com/pantShrey) > > Ideally we would end up with an "push" IPC type writer -- like someone asks "encode this array" and you get back a Vec buffers > > Then we could adapt that to async APIs (and various other IO patterns) quite easily > > I think this is basically how the arrow-ipc encoder internally works anyways Agreed. I think this is blocked at the Arrow IPC boundary rather than purely in DataFusion. `StreamWriter` requires `std::io::Write`, so a DataFusion object-store spill backend would still need buffering or a sync-to-async adapter. A native async/low-copy path likely needs an async-friendly or buffer-producing IPC writer in Arrow. Related Arrow issue: https://github.com/apache/arrow-rs/issues/7812 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
