hi Jacques, This is very exciting! LLVM codegen for Arrow has been on my wishlist since the early days of the project. I always considered it more of a "when" question more than "if".
I will take a closer look at the codebase to make some comments, but my biggest initial question is whether we could work to make Gandiva the official community-supported LLVM framework for creating JIT-compiled Arrow kernels. In the Ursa Labs (a new lab I am building to focus 90+% on Apache Arrow development) tech roadmap we discussed the need for a subgraph compiler using LLVM: https://ursalabs.org/tech/#subgraph-compilation-code-generation. I would be interesting in getting involved in the project, and I expect in time many others will, as well. An obvious question would be whether you would be interested in donating the project to Apache Arrow and continuing the work there. We would benefit from common build, testing/CI, and packaging/deployment infrastructure. I'm keen to see JIT-powered predicate pushdown in Parquet files, for example. Phillip and I could look into building a Gandiva backend for compiling a subset of expressions originating from Ibis, a lazy-evaluation DSL system with similar API to pandas (https://github.com/ibis-project/ibis). best Wes On Thu, Jun 21, 2018 at 4:13 PM, Dimitri Vorona <alen...@googlemail.com.invalid> wrote: > Hey Jaques, > > Great stuff! I'm actually researching the integration of arrow and flight > into a main memory database which also uses LLVM for dynamic query > generation! Excited to have a more detailed look at Gandiva! > > Cheers, > Dimitri. > > On Thu, Jun 21, 2018, 21:15 Jacques Nadeau <jacq...@apache.org> wrote: > >> Hey Guys, >> >> Dremio just open sourced a new framework for processing data in Arrow data >> structures [1], built on top of the Apache Arrow C++ APIs and leveraging >> LLVM (Apache licensed). It also includes Java APIs that leverage the Apache >> Arrow Java libraries. I expect the developers who have been working on this >> will introduce themselves soon. To read more about it, take a look at our >> Ravindra's blog post (he's the lead developer driving this work): [2]. >> Hopefully people will find this interesting/useful. >> >> Let us know what you all think! >> >> thanks, >> Jacques >> >> >> [1] https://github.com/dremio/gandiva >> [2] https://www.dremio.com/announcing-gandiva-initiative-for-apache-arrow/ >>
