JingsongLi opened a new pull request, #5:
URL: https://github.com/apache/paimon-vector-index/pull/5
## Summary
- **ivfpq**: IVF-PQ index — the central index structure tying together
kmeans (coarse quantizer), PQ (compression), OPQ (rotation), and fastscan
(4-bit SIMD acceleration)
- Train/add/search pipeline with rayon parallelization, optional ID
filtering, max_codes budget, precomputed tables, and distributed build support
(from_trained + merge_from)
- Supports L2, InnerProduct, and Cosine metrics; both 8-bit and 4-bit PQ
Reader-based search functions (`search_with_reader`, `search_batch_reader`)
are deferred to a follow-up PR with the `io` module.
~1133 lines of new code, all clippy-clean and tested (30 tests pass, 7 new).
## Test plan
- [x] `cargo fmt --all -- --check` passes
- [x] `cargo clippy --all-targets --workspace -- -D warnings` passes
- [x] `cargo test --all` passes (30 tests, 7 new ivfpq tests)
- test_build_and_search_l2: L2 distance build + search, verifies top-1
recall
- test_build_and_search_ip: InnerProduct distance build + search
- test_search_with_filter: ID-filtered search, verifies filter enforcement
- test_batch_search: multi-query batch search, verifies self-recall
- test_4bit_ivfpq: 4-bit PQ mode, verifies recall and ~50% storage
reduction
- test_max_codes_early_termination: bounded scan budget
- test_from_trained_and_merge: distributed build workflow (train once, add
separately, merge)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
