JunRuiLee opened a new issue, #514: URL: https://github.com/apache/paimon-rust/issues/514
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Description ## Motivation Apache Paimon (Java) added **bucket-local vector indexes for primary-key tables** — a mechanism distinct from the global/append vector index that paimon-rust already reads (#410). Primary-key tables maintain a per-bucket ANN payload during compaction; each payload maps a vector to a **physical row position in a source data file**. A search combines ANN hits with an **exact fallback** over every active data file **not covered by any ANN segment's source metadata** (e.g. L0 files and freshly written files — ANN sources are restricted to `COMPACT` files above level 0), filters stale candidates by deletion vectors, merges a deterministic global Top-K, and materializes the selected physical rows. This issue tracks giving paimon-rust the ability to **read** primary-key vector search results — read the on-disk index/data written by Java Paimon and reproduce its search + physical-row materialization on the read side. ## Tracking scope This issue mirrors the Java implementation as of commit `a50a36ff8` (PR #8579, "Search primary-key vector indexes"). The primary-key vector feature is still under **active development in Java** (added across #8549, #8562, #8563, #8569, #8576, #8579, with engine wiring like #8581 in progress). This issue will **track the latest Java implementation as it evolves — not only the PRs listed here.** The PR breakdown below is the current best plan and will be revised as the Java side lands follow-ups or adjusts contracts. Format stability so far: the **payload + source-metadata on-disk format** is fixed by #8549 (`_SOURCE_META` / `PkVectorSourceMeta`) and #8562 (ANN segment / index-manifest coexistence rules); the **read-side contracts** (exact Top-K semantics, ANN/exact merge, physical-position `IndexedSplit` coordinates) are fixed by #8569 and #8576. Recent PRs declare "no storage-format changes", but this will keep being re-verified against Java `master`. paimon-rust already has a read-only vindex reader (`crates/paimon/src/vindex/`) that performs ANN search, so PR2 reuses it. **Key interop assumption / risk**: ANN payload bytes are written by Java's `VectorGlobalIndexer`, so this work depends on those bytes being readable by the paimon-vindex-core / Lumina FFI reader — the same assumption #410 already relies on. This also bounds which index types (e.g. `ivf-pq`) PR2 can actually validate against Java fixtures. ## Scope (incremental, read-only) - [ ] **PR 1** — Parse ANN source metadata (`_SOURCE_META` / `PkVectorSourceMeta` / `PkVectorSourceFile`) off `GlobalIndexMeta.sourceMeta`, and map a segment ordinal to `(data file, physical row position)`. (Java #8549) - [ ] **PR 2** — Bucket-local search: ANN (via the existing `vindex` reader) + exact fallback over uncovered active files + distance metrics (`l2` / `cosine` / `inner_product`, bounded-memory exact Top-K) + deletion-vector live-row filtering. (Java #8569) *(Unit-testable in isolation with synthetic per-bucket inputs; real planning arrives in PR4.)* - [ ] **PR 3a** — Physical-position single-file reader that reads selected bucket-local row positions from one data file and exposes search scores. This is a **new component** ported from Java's `PrimaryKeyVectorPositionReader` / `PrimaryKeyIndexedSplitRead`; rust has no equivalent (existing raw reads use global row-id ranges + sort-merge). (Java #8576) - [ ] **PR 3b** — Global Top-K merge across buckets and materialization of the selected physical rows. (Java #8579) - [ ] **PR 4** — Snapshot-consistent planning (group data files, deletion vectors, and payloads per `(partition, bucket)`) and end-to-end read validation against fixtures written by Java Paimon. (Java #8579) ## Out of scope (tracked separately) - **Writing / building** primary-key vector indexes — Java builds these during primary-key compaction, which paimon-rust does not yet do. - **Engine SQL entry points** (Spark #8581 / Flink) — paimon-rust exposes a read/search API rather than mirroring the engine SQL layer. - Hybrid search, batch / lateral vector search. ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
