JunRuiLee opened a new pull request, #550: URL: https://github.com/apache/paimon-rust/pull/550
### Purpose Part of #514. Adds **exact rerank** to primary-key vector search, mirroring Java `PrimaryKeyVectorRead`'s refine/rerank. On a PK-vector table, `execute_read()` materializes rows in file-local physical coordinates; this lets a configured refine factor over-fetch approximate (ANN) candidates and reorder them by exact distance for higher recall, while leaving the no-refine path byte-identical. Stacked on #537 (C FFI materialized read); its two commits appear in this diff until #537 merges. Review/merge after #537. ### Brief change log - **Dual-limit bucket search**: `bucket_search` returns separate indexed/exact candidate lists, each bounded independently — the ANN/indexed channel over-fetches (`indexed_limit = limit × refine_factor`) while the exact-fallback channel stays bounded by the final `limit`. `PkVectorOrchestrator::search_candidates` takes the dual limits and returns an `OrchestratorSearchResult`; `merge_candidates` runs a global Top-K over the union of the two channels. - **Rerank**: when a positive refine factor is configured, `rerank_indexed_positional` rereads the over-fetched ANN candidates' vectors from their data files by file-local position (reusing the position read path), recomputes exact distance, and re-runs the global Top-K. Exact-fallback candidates are already exact and are not reranked. - **Refine-factor resolution** reuses the existing option-key precedence (`refine_factor` / `refine-factor` / `rerank_factor` / `rerank-factor`, with `fields.<col>.` → index-type → `ivf.` → bare prefix order); factor 0 (unset) leaves `indexed_limit == limit`, so output is byte-identical to the no-rerank path. - **Fail-loud validation**: `append_batch_vectors` rejects null child elements in vector arrays (never a defaulted 0.0), `build_indexed_splits` validates `split_index` is in range before indexing, and the rerank kernel fails loud on ownership/dedup/NULL/dimension/file-not-in-plan violations — never a panic or a silently corrupted result. ### Tests - Dual-limit bucket behavior; `merge_candidates` union Top-K; rerank kernel (ownership order, dedup, NULL element, dimension mismatch, file-not-in-plan); end-to-end refine integration matching an exact brute-force ground truth. - **Regression invariant**: `refine_factor == 0` preserves the exact pre-rerank output (baseline + Java-fixture read-back tests unchanged). - Fail-loud coverage: null vector element, out-of-range `split_index`. - `cargo test -p paimon` green (1803 tests); `cargo build -p paimon-datafusion` clean (cross-crate `Send` gate); `cargo clippy -D warnings` and `cargo fmt --check` clean. ### API and Format No on-disk format change, no new result columns. `refine_factor == 0` (the default) is byte-identical to the current output. ### Documentation Code comments only; no user-facing docs change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
