CrownChu opened a new pull request, #59: URL: https://github.com/apache/paimon-vector-index/pull/59
## What is changed - Stop filtered HNSW cardinality checks as soon as the exact-scan threshold is exceeded. - For single-query IVF-HNSW-SQ reader searches, decode row IDs and SQ codes first and defer SQ-vector/HNSW graph materialization until graph search is actually required. - Extend the existing filter benchmark with a `single` mode. - Add threshold boundary/short-circuit tests and reader coverage for the broad-filter HNSW fallback. The selective path already uses an exact SQ scan when the number of matching IDs is at most `max(k, ef_search)`. Previously the reader materialized every selected HNSW graph before making that decision. There are no public API, dependency, or on-disk format changes. Unfiltered and batch reader paths are unchanged. ## Benchmark Baseline: `26044a3` Candidate: `24844c5` ```bash FILTER_BENCH_N=50000 FILTER_BENCH_NQ=500 FILTER_BENCH_D=128 FILTER_BENCH_K=10 FILTER_BENCH_NLIST=64 FILTER_BENCH_NPROBE=32 FILTER_BENCH_EF_SEARCH=80 FILTER_BENCH_FILTER_STRIDES=256,512,1024 FILTER_BENCH_SEARCH_MODE=single cargo bench -p paimon-vindex-core --bench ivfhnswsq_filter_bench -- --nocapture ``` | Filter stride | Allowed IDs | Baseline search | Candidate search | Speedup | | ---: | ---: | ---: | ---: | ---: | | 256 | 196 | 5119 ms | 2535 ms | 2.02x | | 512 | 98 | 5066 ms | 2512 ms | 2.02x | | 1024 | 49 | 4775 ms | 906 ms | 5.27x | A broad-filter control run (`NQ=200`, strides `1,4`) was also neutral: `2190 -> 2159 ms` and `2159 -> 2136 ms`. The benchmark checks that results before and after `optimize_for_search` are identical. ## Validation - `python3 tools/validate_asf_yaml.py` - `cargo fmt --all -- --check` - `cargo clippy --all-targets --workspace -- -D warnings` - `cargo build --all-targets` - `cargo test --workspace` - Selective and broad-filter benchmark runs shown above -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
