JunRuiLee opened a new pull request, #108: URL: https://github.com/apache/paimon-vector-index/pull/108
## Dependency and review baseline **Depends on #107; please keep this PR as a draft until that dependency is merged.** This branch starts at #107's head, `907072dfefd9b075ecfeea3377665c5688a0fa4f`, and adds one cosine/inner-product extension commit, `fb87cda`. - [Extension-only diff against #107](https://github.com/JunRuiLee/paimon-vector-index/compare/907072d...fb87cda): 18 files, +1,675 / -245 lines. More than 75% of the additions are integration tests and documentation. - Because #107 is still open, the comparison against upstream `main` also includes its IVF-PQ L2 range-search commit. That parent implementation is not a new implementation in this extension. - After #107 merges, rebase this branch onto `main` to leave only the metric extension in the PR diff. ## Summary - Support cosine and inner-product range search for IVF-Flat, IVF-SQ, IVF-PQ and IVF-RQ through all four typed/unified single, batch and Roaring-filtered entry points. - Define shared public score conversion, exact f64 predicate-to-f32 band conversion, IP side/operator reversal, signed-zero/extreme endpoint handling, and range capability queries. - Preserve the membership contract: Flat uses exact stored-vector distances; SQ/PQ/RQ use their declared estimates. Full probing does not remove quantization error. - Extend #107's native PQ scanner rather than adding a separate scanner. Preserve PQ4/PQ8, residual coding, OPQ, lazy bounded LUT caching, scratch fallback, parallel query scans, shared filtering and bounded streaming. - Normalize cosine queries before probing/scanning (and before PQ OPQ). PQ cosine uses half the floating-point ADC squared-distance estimate; IP uses negative estimated dot products with residual coarse contributions included once. No cosine/IP partial-sum pruning is used. - Add oracle, ULP/boundary, nonfinite-data, zero-vector, filter, statistics, cache/streaming and L2/top-K regression coverage, and update the Rust/docs capability and estimator descriptions. Existing L2 and top-K behavior, the storage format and C/JNI APIs remain unchanged. No original-vector reranking or top-K fallback is introduced. ## Validation Run locally on macOS/AArch64: - `cargo test --offline --locked --workspace`: **656 passed, 2 ignored**. - `cargo test --offline --locked --release --workspace -- --test-threads=1`: **656 passed, 2 ignored**. This serializes the test harness, not the explicit 1/4-worker Rayon pools exercised by the tests. - `cargo clippy --offline --locked --workspace --all-targets -- -D warnings`: passed. - `cargo fmt --all -- --check` and `git diff --check`: passed. - Local documentation links/anchors: 192 references passed across six changed HTML pages. - PQ coverage includes all metrics, PQ4/PQ8, both residual modes, dense 64D OPQ, scalar-oracle comparisons, adjacent-ULP membership, single/batch/filter equivalence, 1/4 workers and `optimize_for_search` invariance. Streaming crosses the production 8 MiB LUT cache budget for PQ8; PQ4 scratch fallback is covered separately with a forced budget. Local independent read-only agent reviews of shared semantics/Flat/SQ/RQ and the PQ extension found no remaining in-scope blockers. Maintainer review and hosted CI are still pending. ## Known validation limitation Default-concurrent Release test runs intermittently fail unchanged top-K tests that assert work was observed on multiple Rayon workers. A clean archive of #107, compiled into a separate target directory, reproduced the same category of IVF-Flat worker-count assertion failure. Isolated tests and the complete serialized Release test harness pass. This is not a claim that default-concurrent Release is consistently green. The expanded SQ range parallelism test uses a bounded worker handshake to avoid that scheduling race in its own coverage; unrelated top-K tests are not changed here. ## Rollback / follow-up No data migration or format change is involved. Reverting only the extension commit leaves #107's L2 implementation intact. Before marking ready: merge #107, update this branch onto `main`, and inspect hosted CI and maintainer feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
