Thanks for all the updates. I'd like to get involved and help out with this effort as well. I don't have any major work planned for DataFusion for 1.0.0 now other than maybe moving to the new parquet ArrowReader, if it is ready in time.
I have been chatting with the author of the Rust Flatbuffer project about some of the issues and I can take an action to follow up with that. I have also been talking with one of the authors of Tonic, and I believe they might be interested in helping here too. Let me know how else I can help out with this effort. Andy. On Thu, Oct 17, 2019 at 2:42 PM Neville Dipale <nevilled...@gmail.com> wrote: > Good evening > > With support for testing against integration files now done, I've resumed > work on the IPC reader. If I don't encounter trouble reading the existing > files, I expect to be done with this work by the end of the weekend. I had > taken the approach of one large PR to include all Rust-supported Arrow > types. > > I'm not sure of how long the writer would take, but I remain committed to > having this work completed by 1.0. > > I have to catch up on null-type roundtrip and the padding alignment work as > I haven't been able to keep abreast with development these last few months. > Also, the Rust flatbuffer issues that we've had haven't progressed in the > relevant repo, so it still makes ergonomics not great, but at least our > users don't have to worry about that. > > @Andy I have good experience with gRPC in Rust, and also want to see Flight > support landing soon. > > On Thu, 17 Oct 2019, 17:06 David Li, <li.david...@gmail.com> wrote: > > > Just for reference, it's possible once the basic IPC support is merged; I > > had a proof of concept, though it needs to be updated to use Tonic over > > tower-grpc, actually implement the zero-copy optimizations, provide a > real > > API, etc. > > > > https://github.com/apache/arrow/pull/4167#issuecomment-529695811 > > > > On Thu, Oct 17, 2019, 10:51 Wes McKinney <wesmck...@gmail.com> wrote: > > > > > I hope to see Flight in all the reference implementations eventually. > > > > > > Having hardened IPC support is a pre-requisite, it would be ideal to > > > have Rust as a participant in the integration tests > > > > > > On Thu, Oct 17, 2019 at 9:41 AM Andy Grove <andygrov...@gmail.com> > > wrote: > > > > > > > > I was approached directly about adding Flight support to the Rust > > > > implementation, and said I would start a discussion here on the > mailing > > > > list. > > > > > > > > There is ongoing work with IPC and integration and I believe that it > > > would > > > > make sense to start looking at adding Flight support. > > > > > > > > I'd like to hear what others think though. > > > > > > > > Thanks, > > > > > > > > Andy. > > > > > >
