A (very belated) follow up to this thread -- there is now a POC PR from Raphael Taylor-Davies with an async Arrow reader for parquet files [1] if anyone is interested.
Andrew [1] https://github.com/apache/arrow-rs/pull/1154# On Wed, Aug 26, 2020 at 12:52 PM Andy Grove <andygrov...@gmail.com> wrote: > Hi Max, > > I have been experimenting with an async record batch reader and was able to > get a working version, but I had to use channels to communicate with the > parquet reader, which ran on its own thread. > > I have taken a step back now that I have some experience of this and look > forward to working with others on a design for async. I know that there are > some other contributors who are interested in this as well. > > I can take an action to start a Google doc where we can discuss this. I > think this will be useful while we all align on what problems we are trying > to solve and then we can take the findings back to JIRA. I'll create a doc > and reply to this thread in the next few days. > > Here's the JIRA issue that Mahmut was referring to: > > https://issues.apache.org/jira/browse/ARROW-9275 > > There are some related issues as well, but without much detail: > > https://issues.apache.org/jira/browse/ARROW-9674 > https://issues.apache.org/jira/browse/ARROW-9754 > > Thanks, > > Andy. > > > On Wed, Aug 26, 2020 at 10:31 AM Vertexclique <vertexcli...@gmail.com> > wrote: > > > Hi Max; > > > > There is an open issue in the tracker which needs to gather feedback to > > finalize how we will do overall async interface which spans to arrow > > crates. Please check that issue, it is mentioning sans IO and several > > design considerations. Imo we can carry async discussion under it. > > > > Best, > > Mahmut > > > > On Aug 26, 2020, 19:18, at 19:18, Max Burke <m...@urbanlogiq.com> wrote: > > >Out of curiosity, is anyone working on a record batch reader that's > > >async > > >friendly? > > > > > >Wanting to know if it's something I could wait on/help out with, or if > > >it's > > >something we could start working on too. > > > > > >-- > > >-Max > > >
