Daniël Heres created ARROW-11112:
------------------------------------
Summary: [Rust][DataFusion] Implement vectorized hashing
Key: ARROW-11112
URL: https://issues.apache.org/jira/browse/ARROW-11112
Project: Apache Arrow
Issue Type: Improvement
Components: Rust - DataFusion
Reporter: Daniël Heres
Currently, the approach of the join and hash aggregates is to create a key
individually from the row values. However, this is far from ideal, as it
doesn't utilize the cache vectorized nature of Arrow, but instead copies data
into a vec, traverses multiple arrays in the inner loop, etc.
This blog post has a summary of an approach to do this in a vectorized way.
[https://www.cockroachlabs.com/blog/vectorized-hash-joiner/]
TBD:
We should decide/find out whether it still makes sense to use rust `HashMap`
(with () as key?) or whether to create an own? Benefit of using hashmap is that
there is an API, can resize automatically, and uses SIMD, and also exposes some
lower level bits we can use here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
