I'm a fan of using Rayon. Perhaps if it's not too much work, we could compare the two
On Mon, 4 Mar 2019 at 15:04, Krisztián Szűcs <szucs.kriszt...@gmail.com> wrote: > On Mon, Mar 4, 2019 at 5:55 AM Andy Grove <andygrov...@gmail.com> wrote: > > > I have been working on a PoC of parallel query execution and it is > working > > well, and I am now starting to create PRs for the various refactors > > necessary for this in DataFusion. > > > > I haven't been following the async/await and futures/tokio developments > > lately but for the PoC I used tokio-threadpool which seems simple to use. > > > > I just wanted to give everyone a chance to give their thoughts on this > > before I get too far with my batch of PRs. Is anyone opposed to using > > tokio-threadpool? > > > DataFusion's tasks should be CPU bound and according to tokio-threadpool's > documentation [1], it is more suitable for event loops: > "It is optimized for the primary Tokio use case of many independent tasks > with limited computation and with most tasks waiting on I/O." > > Rayon seems to follow different semantics, but depending on futures-rs is > considerable, especially because it is maintained by the rust lang nursery. > > [1] https://docs.rs/tokio-threadpool/0.1.12/tokio_threadpool/ > > Cheers, Krisztian >
