JingsongLi opened a new pull request, #131: URL: https://github.com/apache/paimon-vector-index/pull/131
## Motivation Paimon can avoid opening every row-range shard for a nearest-neighbor query when builders and readers agree on a portable coarse model. This PR adds the Rust and Java/JNI operations needed to train that model once, route rows on other workers and search just the selected shard files. The original IVF-PQ design is described in [PIP-47](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=444334660). ## API and storage - IVF-FLAT, IVF-SQ, IVF-PQ and IVF-RQ share the trained quantizer across writers. Centroid training uses exact coarse assignment so residual training, row assignment and query routing agree. DiskANN shares a coarse router and builds a local PQ model and graph for each populated shard. - A worker can restore a serialized model, derive independent writers, append pre-routed absolute row IDs, and use single or batched shard search with an optional Roaring filter. The caller maintains the centroid-to-file mapping and merges shard results. - Shard files use the existing family-specific format. Versioned, data-free training envelopes have frozen fixtures and metadata limits. Size preflight accounts for replicated training state and per-graph page alignment. - Java applications using an external JNI library should deploy it alongside the matching JAR. Rust callers constructing the low-level `DiskAnnIndex` with struct literals need to use its constructors. Paimon table integration is maintained in the Paimon repository. Related work on reusable ordinary writers is tracked in [PR #87](https://github.com/apache/paimon-vector-index/pull/87). ## Performance evidence The following SIFT1M measurements come from the candidate **before the final review fixes**. Each cell shows range → centroid. The benchmark used 1M 128-dimensional L2 vectors, 2,000 queries per comparison, top-10 and a fresh reader per query on warm OS page cache. Build and P95 are medians of three measured rounds. Reads are mean logical MiB per query. | Index | Recall@10 | Build, s | P95, ms | Files/query | MiB/query | | --- | --- | --- | --- | --- | --- | | IVF-FLAT | 0.9900 → 0.9911 | 2.21 → 4.39 | 9.09 → 5.00 | 64 → 21 | 67.18 → 43.25 | | IVF-SQ | 0.9802 → 0.9826 | 2.43 → 2.86 | 9.50 → 4.61 | 64 → 11 | 22.54 → 21.34 | | IVF-PQ | 0.7307 → 0.7371 | 65.60 → 19.12 | 30.26 → 1.45 | 64 → 3 | 14.98 → 2.19 | | IVF-RQ | 0.9008 → 0.9084 | 4.35 → 11.08 | 29.10 → 12.13 | 64 → 19 | 13.04 → 8.54 | | DiskANN | 0.9946 → 0.9944 | 95.20 → 103.31 | 45.15 → 10.72 | 64 → 11 | 82.48 → 16.83 | The matched-recall rule was an absolute mean difference within one percentage point and a paired 95% bootstrap lower bound above −1 point. IVF-PQ uses the frozen original query cohort for this comparison and replays identical per-query hit counts. Its centroid configuration uses 64 PQ subquantizers vs. 32 for range, increasing index size from 41.7 to 76.2 MiB. On two additional cohorts its recall advantage exceeded the matching band. FLAT and RQ also trade extra build time and footprint for fewer reads. ## Verification on the reviewed code - Ran the Rust workspace tests in release mode, workspace Clippy with warnings denied, and formatting checks. - Verified all five frozen model fixtures, routed single/batch/filtered results, query error statistics, reader metadata limits and compact/interleaved DiskANN storage budgets. - Built and verified the macOS arm64 JAR under JDK 8, checked its native library against the release build, and ran the native loader, validation and handle-safety programs from the packaged JAR. - Replayed the IVF-PQ recall benchmark through Cargo on 10,000 vectors and 64 queries at probe widths 1, 4 and 16. Training and restoring the same model produced byte-identical model files, index files and per-query results. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
