JunRuiLee opened a new pull request, #517: URL: https://github.com/apache/paimon-rust/pull/517
### Purpose Linked issue: part of #514 — PR 3a of the primary-key vector read series (does not close the issue) > **Stacked on #516 (PR 2), which is stacked on #515 (PR 1).** This branch is based on the PR 2 branch, so its commit range currently **includes PR 1's and PR 2's commits**. Please review only the commits/files added by this PR (the `table/pk_vector_position_read.rs` module + three `DataFileReader` accessors). Once #515 and #516 merge, I will rebase this branch onto `main` and the diff will collapse to PR 3a's changes only. This ports the **reader-kernel subset** of Java apache/paimon#8576 (`[core] Read primary-key vector results by position`) as of commit `a50a36ff8` — the Rust equivalent of `PrimaryKeyVectorPositionReader`. Given one data file, a set of selected 0-based physical row positions, and an optional position->score map, it materializes the selected physical rows and exposes, per row, its physical position and aligned score. Positions are recovered from the actual rows the lower-level read emits (post deletion-vector filtering), so alignment survives DV-deleted rows. Read-only. Out of scope for this PR (tracked in #514, split from Java #8576): the `IndexedSplit` read-path contract / single-file validation / TopN-limit pushdown disabling (PR 3b), cross-bucket Top-K merge + materialization orchestration (PR 3c), snapshot-consistent planning + end-to-end validation against Java-written fixtures (PR 4), and projection cleanup / final user-facing column selection (PR 3b). ### Brief change log New crate-private component `crates/paimon/src/table/pk_vector_position_read.rs`: - `PkVectorPositionRead<'a>` wraps `&DataFileReader` and exposes `read(split, file_meta, data_fields, dv, positions, scores)`, returning a `RecordBatch` stream. - **Rust expression of Java's per-row `ScoreRecordIterator` side-channel:** since the Rust read path is a `RecordBatch` stream with no per-row side-channel, the reader appends internal reserved metadata columns instead — `_PKEY_VECTOR_POSITION` (`Int64`, always) and `_PKEY_VECTOR_SCORE` (`Float32`, only when scores are provided). These are established here as a contract the later read-path PRs consume. - **Physical positions are derived from the `_ROW_ID` column** the existing selective read emits (`position = _ROW_ID - first_row_id`), not from batch offsets or requested positions — so a deletion vector dropping rows never desyncs position/score alignment. `_ROW_ID` is injected internally into a cloned reader and removed from the output. - Reuses the existing selective single-file read (`DataFileReader::read_single_file_stream`, its `_ROW_ID` mechanism, and DV intersection) rather than a new read path. - Validation (all `DataInvalid`): empty positions rejected (matching Java `checkArgument(!rowPositions.isEmpty())`); negative / out-of-range positions; scores key-set must exactly equal the selected positions; a requested `_ROW_ID` or a reserved metadata column name is rejected; a predicate-bearing reader is rejected (a **local** guard because `_ROW_ID` + a row-filtering predicate desync — the final indexed-read predicate policy is PR 3b's); a data file without `first_row_id`. - Adds three `pub(super)` accessors to `DataFileReader` (`read_type`, `has_row_filtering_predicate`, `with_read_type`) for the sibling module; `has_row_filtering_predicate` treats a lone `AlwaysTrue` as non-filtering, matching the existing `_ROW_ID`/predicate guard. - The module and the three accessors carry a temporary `#[allow(dead_code)]` (no production caller until PR 3b wires them); removed when PR 3b lands. ### Tests All synthetic (in-memory parquet via the existing `data_file_reader` test helpers); 16 tests in the module, full `cargo test -p paimon` green; `cargo fmt --all --check` and `cargo clippy -p paimon --all-targets -- -D warnings` both clean: - basic selected-position read (position column values, ascending, no `_ROW_ID`/`_SCORE` leak); non-contiguous score alignment `[0,2,5] -> [0.9,0.5,0.1]`; score omitted -> no score column; deletion-vector skips positions while keeping alignment (`select [0,1,2,3]`, delete 1 -> positions `[0,2,3]`, discriminating against a batch-offset implementation); multi-batch alignment across a batch boundary. - error branches: empty positions, negative position, out-of-range position, scores key mismatch, requested `_ROW_ID`, reserved column-name conflict, predicate-bearing reader, missing `first_row_id`. - `positions_to_global_ranges` coalescing + `first_row_id` offset, and its `checked_add` overflow rejection. Real ANN vector-search correctness is intentionally **not** exercised here — deferred to PR 4's Java-written fixtures. ### API and Format - No storage-format change. All additions are crate-private (`pub(crate)`/`pub(super)`); no public API and no existing API changed. ### Documentation No user-facing documentation changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
