839224346 opened a new pull request, #9315: URL: https://github.com/apache/paimon/pull/9315
### Purpose Optimize pypaimon's vector raw search path by replacing the pure-Python loop-based distance computation with numpy vectorized operations, and improve index split concurrency using `ThreadPoolExecutor`. **Key changes:** 1. **Numpy vectorized distance computation** — Replace per-row Python loop (`_compute_score` + heap) with batch matrix operations (`_raw_search_from_arrow` + `_numpy_topk`), leveraging Arrow's zero-copy buffer for direct numpy matrix construction. 2. **O(n) top-K selection** — Use `np.argpartition` instead of a heap-based approach, reducing top-K selection from O(n·log k) to O(n). 3. **ThreadPoolExecutor for index splits** — Replace the old `wait(futures)` pattern with `ThreadPoolExecutor` + `as_completed`, adding synchronous `_eval_sync` / `_eval_batch_sync` methods that properly manage reader lifecycle with try/finally. Single-split case avoids thread pool overhead entirely. 4. **Standalone benchmark script** — Added `benchmark_vector_search_standalone.py` for reproducible performance validation without requiring a running Paimon table. **Why:** The raw search path is a fallback for data that hasn't yet been indexed by Faiss/HNSW (e.g., newly written data before compaction). Previously, this path used a pure-Python loop iterating row by row — acceptable for small datasets but extremely slow at scale. The ThreadPoolExecutor change also fixes a subtle issue: the old `_eval()` returned futures with reader references via callbacks, but the reader lifecycle wasn't guaranteed in error paths. The new `_eval_sync` uses explicit try/finally. ### Tests ```bash # Standalone benchmark (no Paimon table required) python3 pypaimon/tests/benchmark_vector_search_standalone.py --num-rows 100000 --dim 768 # Existing test suite python -m pytest pypaimon/tests/ -q -k "vector" # Static analysis python -m pyflakes pypaimon/table/source/vector_search_read.py git diff --check ``` **Performance results (768 dimensions, top-100, isolated processes):** Environment: 16-core CPU, 32GB RAM, numpy 2.x + OpenBLAS 0.3.34 (Haswell, 64-bit int) Each path runs in its own process to avoid memory contention — matches production behavior. 100K rows: | Metric | Python loop | Numpy (fast path) | Speedup | |--------|------------:|-------------------:|--------:| | L2 | 4,449 ms | 695 ms | **6.4×** | | Cosine | 6,214 ms | 607 ms | **10.2×** | | Inner Product | 4,908 ms | 400 ms | **12.3×** | 500K rows: | Metric | Python loop | Numpy (fast path) | Speedup | |--------|------------:|-------------------:|--------:| | L2 | 23,997 ms | 3,318 ms | **7.2×** | | Cosine | 32,129 ms | 2,229 ms | **14.4×** | | Inner Product | 21,696 ms | 1,862 ms | **11.7×** | Top-K correctness: 100% overlap between Python loop and numpy path across all metrics. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
