tustvold commented on issue #3620: URL: https://github.com/apache/arrow-rs/issues/3620#issuecomment-1406951690
One thing to perhaps be aware of is that we are entirely reliant on LLVM to vectorise various kernels. I have found that _any_ branch in the body of the loop, even something as simple as `Iterator::next`, causes it to fail to vectorise the loop. This has led to a number of tricks such as `PrimitiveArray::unary` and `MutableBuffer::collect_bool` that help it vectorise correctly, by ignoring the null mask. I suspect a selection vector would run into similar challenges. Whilst I don't doubt that custom SIMD kernels could apply selection masks and null masks "for free", I have not found a good way to get LLVM to figure this out. As such I wonder if a first step might be to implement this solely within DataFusion, in particular using the `filter` kernel to eagerly apply the selection to the values if the next expression is not side-effect free, or if selectivity has fallen below some threshold? We can then go from there if this filtering starts to become a major bottleneck? This would also allow us to potentially investigate less intrusive changes, depending on where the filter overheads manifest. For example, late materialization of this form was recently added to the parquet reader. We could theoretically build upon this base, without needing to make more intrusive modifications. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
