JunRuiLee commented on issue #514:
URL: https://github.com/apache/paimon-rust/issues/514#issuecomment-4955251781
### Roadmap update — add PR 5 (optional): exact rerank
Java added exact reranking to primary-key vector search in
apache/paimon#8591. Recording it here so the roadmap stays current.
**What it is:** after ANN returns approximate candidates, over-fetch by a
*refine factor*, reread the candidates' original vectors **by physical
position**, recompute exact distances, then do the global Top-K. Config-gated:
disabled when no refine option is set; a factor of `1` recomputes exact scores
without over-fetching. Java's PR explicitly states it **does not modify table,
index, or result formats** — so this does not affect the on-disk format
assumptions of PR 1.
**Impact on this roadmap:** it does **not** change the scope of PR 1–4.
Rerank composes on top of primitives we're already building:
- "reread original vectors by physical position" = **PR 3a**'s position
reader kernel
- "recompute exact distances" = **PR 2**'s metric semantics
- refine-config resolution already exists on the paimon-rust
*data-evolution* vector path (`table/vector_search_builder.rs`); Java #8591's
core idea is to **share that refine config between the data-evolution and
primary-key paths**, which mirrors cleanly.
So it becomes a new, **optional, downstream** slice rather than a rewrite of
any existing PR.
- [ ] **PR 5 (optional) — Exact rerank for PK vector search** ←
apache/paimon#8591
Over-fetch ANN by refine factor → reread by physical position (reuses PR
3a) → recompute exact distance (reuses PR 2) → global Top-K. Config-gated (off
without a refine option; factor=1 = recompute, no over-fetch). No format
change. Depends on PR 3b/PR 3c. Shares the existing data-evolution refine
config plumbing with the PK path.
**Priority:** intentionally last / optional. Without rerank, PK vector
search is still end-to-end usable (ANN + exact fallback); rerank is a
recall/precision enhancement that only triggers when a refine option is
configured. Will be picked up after PR 3b/3c/4 land, if a real recall need
exists. Anchor: apache/paimon#8591 (mirror its merge commit once merged).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
