yjshen opened a new issue #1850: URL: https://github.com/apache/arrow-datafusion/issues/1850
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** With JIT codegen, we could generate specific code for each query to reduce branching overhead from the generalized interpret mode execution. Furthermore, we could reduce the memory footprint during the execution by chaining multiple Arrow compute kernels together and reusing the intermediate vectors. The row format we introduced lately in #1782 is another motivating use case for JIT code generation. We could eliminate most branching during row to batch or batch to row conversion since the branching pattern is set by schema, enjoy the potential advantage of several use cases for row format, and pay the least price during format conversion. **Describe the solution you'd like** 1. Create a generalized code constructing API that could fulfill our needs on expressing query-specific exec functions. 2. Utilize [Cranelift Code Generator](https://github.com/bytecodealliance/wasmtime/tree/main/cranelift), which is stable enough and can fulfill our use cases currently, eliminating branch and glue code fragments into runnable functions. **Describe alternatives you've considered** Build code generation runtime over LLVM instead of Cranelift. However, I think LLVM is over-killing since most of the performance-intensive computations are done at the Arrow kernel. **Additional context** We also have an item in our Roadmap: "Push-based query execution and code generation." -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
