mikemccand commented on code in PR #16723:
URL: https://github.com/apache/lucene/pull/16723#discussion_r4185236640
##########
.github/workflows/run-special-checks-sandbox.yml:
##########
@@ -66,6 +66,12 @@ jobs:
- name: Prepare Lucene workspace
uses: ./.github/actions/prepare-for-build
+ - name: Prepare AVX2 Faiss C library
+ run: |
+ cp "$CONDA_PREFIX/lib/libfaiss_c.so"
"$CONDA_PREFIX/lib/libfaiss_c_avx2.so"
+ patchelf --replace-needed libfaiss.so libfaiss_avx2.so
"$CONDA_PREFIX/lib/libfaiss_c_avx2.so"
Review Comment:
Any other specific SIMD version optimized Faiss binaries ship as well?
##########
lucene/sandbox/src/java/org/apache/lucene/sandbox/codecs/faiss/FaissNativeWrapper.java:
##########
@@ -37,8 +38,44 @@
*/
@SuppressWarnings("restricted") // uses unsafe calls
final class FaissNativeWrapper {
- static {
- System.loadLibrary(FaissLibrary.NAME);
+
+ static final String LOADED_LIBRARY = loadLibrary();
+
+ private static String loadLibrary() {
+ // Manual override via system property
+ String customLib = System.getProperty("lucene.faiss.libname");
+ if (customLib != null && customLib.isBlank() == false) {
+ System.loadLibrary(customLib);
+ return customLib;
+ }
+ // Try SIMD variants matching the CPU architecture
+ switch (Constants.OS_ARCH) {
+ case "amd64", "x86_64" -> {
+ if (Constants.HAS_AVX512 && tryLoad("faiss_c_avx512")) {
Review Comment:
Could we `LOG.warning` here if the CPU supports `AVX512` but Faiss
AVX512-optimized binary is not available, and that we are falling back to X?
It's a horrible silent performance trap when users are doing vector search and
don't realize they are using a slow path because they didn't provide the right
native libraries (Faiss and Lucene's builtin / default Codec).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
