JunRuiLee opened a new pull request, #516:
URL: https://github.com/apache/paimon-rust/pull/516
### Purpose
Linked issue: part of #514 — PR 2 of 5 (does not close the issue)
> **Stacked on #515 (PR 1).** This branch is based on
`feat/pk-vector-source-meta`, so its commit range currently **includes PR 1's
commits**. Please review only the commits/files added by this PR (the
`vindex/pkvector/` module + `DeletionVector::is_deleted`). Once #515 merges, I
will rebase this branch onto `main` and the diff will collapse to PR 2's
changes only.
This is the second, read-only slice of primary-key vector (bucket-local ANN)
search support. It mirrors Apache Paimon (Java) apache/paimon#8569 as of commit
`a50a36ff8`.
It ports the **bucket-local search kernel**: given one snapshot bucket's ANN
segments, active data files, and deletion vectors, merge ANN hits with an exact
fallback over the files not covered by any ANN segment into a deterministic
bounded Top-K. This is a pure, synthetically-tested kernel — the read-path
wiring and real ANN index-byte validation come in PR 3 / PR 4.
Read-only. Out of scope for this PR (tracked in #514): the physical-position
single-file reader (PR 3a), cross-bucket Top-K merge + physical-row
materialization (PR 3b), snapshot-consistent planning and end-to-end validation
against Java-written fixtures (PR 4), index building, and engine SQL entry
points.
### Brief change log
New crate-private module `crates/paimon/src/vindex/pkvector/`:
- `metric.rs` — `VectorSearchMetric` (`l2` / `cosine` / `inner_product`):
`normalize`/`is_supported` (lowercase + `-`→`_`, **no trim**, matching Java),
parse-once enum, `compute_score` (higher-better), `compute_distance`
(lower-better, squared-L2 with no sqrt, cosine denominator widened to f64 like
Java, zero-norm→0), and `score_to_distance` (l2 `1/score-1`→`inf` at 0, no
clamp).
- `result.rs` — `PkVectorSearchResult { data_file_name, row_position,
distance }`.
- `reader.rs` — `PkVectorReader` sequential-scan trait (null rows return
`false` but still advance the physical position); an in-`#[cfg(test)]`
`ArrayReader` fake.
- `exact.rs` — bounded-memory exact Top-K over one file (BEST_FIRST =
distance ASC, then row_position ASC; `total_cmp` for NaN-safety), with
dimension/limit/finite/row-count validation and a deletion-vector exclusion
predicate.
- `ann.rs` — `PkVectorAnnSearcher` trait (bucket search's ANN dependency,
faked in tests) + a structural `VindexAnnSearcher` composed around an injected
scorer seam; plus the pure helpers `build_live_row_ids` (live-row mask in
segment-ordinal space, offsets guarded by `checked_add`) and `map_ann_results`
(ordinal→`(file, position)` via PR 1's `PkVectorSourceMeta::resolve`, rejects
hits on snapshot-deleted rows, `score_to_distance`, BEST_FIRST sort).
- `bucket.rs` — `bucket_search`: validates ANN source metadata against
active files, skips ANN-covered files in the exact fallback (no rescanning),
forwards deletion vectors and search options, and deterministically merges ANN
+ exact candidates (3-key BEST_FIRST across files).
Plus a small cross-module accessor: `DeletionVector::is_deleted(position) ->
bool` (guards `> u32::MAX` → not deleted).
The module is crate-private and guarded by a temporary module-level
`#![allow(dead_code)]`: PR 2 ports the kernel ahead of the read-path wiring, so
it has no production caller yet. The allow is removed once PR 3 / PR 4 lands
callers.
### Tests
All synthetic unit tests inside the module (30 tests; full `cargo test -p
paimon` green; `cargo fmt --all --check` and `cargo clippy -p paimon
--all-targets -- -D warnings` both clean):
- metric: three-metric distance/score anchors (matching Java
`PkVectorExactSearcherTest`), normalize no-trim, cosine zero-norm→0, cosine
f64-sqrt precision (pinned with an irrational-norm case), score_to_distance
l2→inf.
- exact: three-metric distances, dimension/limit/finite/row-count<0
rejection, null + excluded physical-position preservation (Java parity),
tie-break by smaller row_position, limit truncation.
- ann: `build_live_row_ids` None-when-no-relevant-DV + multi-file-offset
masking; `map_ann_results` ordinal→position mapping + score, out-of-range
ordinal rejection, ANN-hit-on-deleted rejection; adapter composition through
the scorer seam.
- bucket: ANN + exact merge with covered files never re-read (asserted via a
recording factory), pure exact fallback with deletion-vector exclusion,
non-positive limit / duplicate file name / ANN-source row-count mismatch /
segments-without-searcher rejection, cross-file BEST_FIRST tie-break.
- deletion vector: `is_deleted` membership + `> u32::MAX` guard.
Real ANN vector-search correctness (driving the `vindex` reader with genuine
index bytes) is intentionally **not** exercised here — the `vindex` crate is
read-only, so index bytes cannot be synthesized in-process. It is deferred to
PR 4's Java-written fixtures.
### API and Format
- No storage-format change: this consumes the `_SOURCE_META` format defined
by Java apache/paimon#8549 (parsed in PR 1) and reproduces the search semantics
of apache/paimon#8569.
- All additions are crate-private (`pub(crate)`); no public API and no
existing API changed.
### Documentation
No user-facing documentation changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
