yordan-pavlov commented on pull request #7037: URL: https://github.com/apache/arrow/pull/7037#issuecomment-622928732
hi @nevi-me , I would love to have arrow use stable rust. (but I also want high performance) I have been working on some benchmarks which compare performance of filtering of in-memory data implemented in different ways, including for-loop, iterator, arrow without SIMD, arrow with SIMD, etc. I have also done pretty much the same benchmarks for .net core and rust. I hope to find some time to publish those benchmarks in the next few days, but it does look like the rust compiler is very good in auto-vectorization. So far the best performance I have managed to produce with arrow and SIMD (357.53 us) is only about twice as fast as filtering using an iterator without arrow (654.22 us) but is about four times faster than filtering with arrow and loops (1.2704 ms) and many times faster than filtering with arrow using compute::no_simd_compare_op (8.4308 ms). Yes, SIMD is a lot of work, but at the moment gives the best performance. I wonder if the SIMD features could be moved to another library separate from the core arrow library. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
