JingsongLi opened a new pull request, #131:
URL: https://github.com/apache/paimon-vector-index/pull/131

   ## Motivation
   
   Paimon can avoid opening every row-range shard for a nearest-neighbor query 
when builders and readers agree on a portable coarse model. This PR adds the 
Rust and Java/JNI operations needed to train that model once, route rows on 
other workers and search just the selected shard files. The original IVF-PQ 
design is described in 
[PIP-47](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=444334660).
   
   ## API and storage
   
   - IVF-FLAT, IVF-SQ, IVF-PQ and IVF-RQ share the trained quantizer across 
writers. Centroid training uses exact coarse assignment so residual training, 
row assignment and query routing agree. DiskANN shares a coarse router and 
builds a local PQ model and graph for each populated shard.
   - A worker can restore a serialized model, derive independent writers, 
append pre-routed absolute row IDs, and use single or batched shard search with 
an optional Roaring filter. The caller maintains the centroid-to-file mapping 
and merges shard results.
   - Shard files use the existing family-specific format. Versioned, data-free 
training envelopes have frozen fixtures and metadata limits. Size preflight 
accounts for replicated training state and per-graph page alignment.
   - Java applications using an external JNI library should deploy it alongside 
the matching JAR. Rust callers constructing the low-level `DiskAnnIndex` with 
struct literals need to use its constructors. Paimon table integration is 
maintained in the Paimon repository.
   
   Related work on reusable ordinary writers is tracked in [PR 
#87](https://github.com/apache/paimon-vector-index/pull/87).
   
   ## Performance evidence
   
   The following SIFT1M measurements come from the candidate **before the final 
review fixes**. Each cell shows range → centroid. The benchmark used 1M 
128-dimensional L2 vectors, 2,000 queries per comparison, top-10 and a fresh 
reader per query on warm OS page cache. Build and P95 are medians of three 
measured rounds. Reads are mean logical MiB per query.
   
   | Index | Recall@10 | Build, s | P95, ms | Files/query | MiB/query |
   | --- | --- | --- | --- | --- | --- |
   | IVF-FLAT | 0.9900 → 0.9911 | 2.21 → 4.39 | 9.09 → 5.00 | 64 → 21 | 67.18 → 
43.25 |
   | IVF-SQ | 0.9802 → 0.9826 | 2.43 → 2.86 | 9.50 → 4.61 | 64 → 11 | 22.54 → 
21.34 |
   | IVF-PQ | 0.7307 → 0.7371 | 65.60 → 19.12 | 30.26 → 1.45 | 64 → 3 | 14.98 → 
2.19 |
   | IVF-RQ | 0.9008 → 0.9084 | 4.35 → 11.08 | 29.10 → 12.13 | 64 → 19 | 13.04 
→ 8.54 |
   | DiskANN | 0.9946 → 0.9944 | 95.20 → 103.31 | 45.15 → 10.72 | 64 → 11 | 
82.48 → 16.83 |
   
   The matched-recall rule was an absolute mean difference within one 
percentage point and a paired 95% bootstrap lower bound above −1 point. IVF-PQ 
uses the frozen original query cohort for this comparison and replays identical 
per-query hit counts. Its centroid configuration uses 64 PQ subquantizers vs. 
32 for range, increasing index size from 41.7 to 76.2 MiB. On two additional 
cohorts its recall advantage exceeded the matching band. FLAT and RQ also trade 
extra build time and footprint for fewer reads.
   
   ## Verification on the reviewed code
   
   - Ran the Rust workspace tests in release mode, workspace Clippy with 
warnings denied, and formatting checks.
   - Verified all five frozen model fixtures, routed single/batch/filtered 
results, query error statistics, reader metadata limits and compact/interleaved 
DiskANN storage budgets.
   - Built and verified the macOS arm64 JAR under JDK 8, checked its native 
library against the release build, and ran the native loader, validation and 
handle-safety programs from the packaged JAR.
   - Replayed the IVF-PQ recall benchmark through Cargo on 10,000 vectors and 
64 queries at probe widths 1, 4 and 16. Training and restoring the same model 
produced byte-identical model files, index files and per-query results.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to