hpvd commented on issue #13773: URL: https://github.com/apache/pinot/issues/13773#issuecomment-2340233395
just to get an understanding how other projects handle this: for apache lucene, using more SIMD in an easy way is one of the reasons to make java v21 mandatory with the upcoming next major release of lucene (v10) V > ectorization > > Parallelism and concurrency, while distinct, often translate to "splitting a task so that it can be performed more quickly", or "doing more tasks at once". Lucene is continually looking at new algorithms and striving to implement existing ones in more performant and efficient ways. One area that is now more straightforward to us in Java is data level parallelism - the use of SIMD (Single Instruction Multiple Data) vector instructions to boost performance. > > Lucene is using the latest JDK Vector API to implement vector distance computations that result in efficient hardware specific SIMD instructions. These instructions, when run on supporting hardware, can perform floating point dot product computations 8 times faster than the equivalent scalar code. This [blog](https://www.elastic.co/blog/accelerating-vector-search-simd-instructions) contains more specific information on this particular optimization. > > With the move to Java 21 minimum, it is a lot more straightforward to see how we can use the JDK Vector API in more places. We're even experimenting with the possibility of calling customized SIMD implementations with FFI, since the overhead of the native call is now quite minimal. https://www.elastic.co/search-labs/blog/lucene-and-java-moving-forward-together -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
