JunRuiLee opened a new pull request, #544:
URL: https://github.com/apache/paimon-rust/pull/544
### Purpose
Linked issue: part of #514
Rust's primary-key (PK) vector read assumed every PK-vector data file
carries a `first_row_id`, using it to turn physical `(file, row position)`
coordinates into global row ids. But Java never assigns `first_row_id` to
PK-table data files — row-tracking (the only thing that assigns it) is
forbidden on PK tables. Java's PK vector search works entirely in physical
coordinates and deliberately does not produce global row ids:
`PrimaryKeyScoredResult.results()` throws, and the physical position reader
selects rows by file-local positions via `batch.selection(rowPositions)`.
As a result, on a real Java-written PK-vector table (no `first_row_id`) both
the scored path and the materialized read fail loud in Rust. The existing tests
only passed because they pinned `first_row_id == 0`, making `first_row_id +
position == position`.
This PR aligns the Rust PK vector read to Java's physical-coordinate model.
### Brief change log
- PK `execute_scored()` now fails loud when the query targets a PK-vector
column (mirrors Java `PrimaryKeyScoredResult.results()` throwing): the PK path
produces physical positions, not global row ids, so callers must use
`execute_read()`. Removed the global-id helper that computed `first_row_id +
position`.
- Reworked the PK materialization read to select rows by **file-local
position** (mirroring Java `batch.selection(rowPositions)`) instead of
round-tripping through a global `_ROW_ID`/`first_row_id`. Added a
local-selection read entry on `DataFileReader` that feeds file-local ranges
straight to the format reader's row-selection, bypassing the global→local
conversion.
- `_PKEY_VECTOR_POSITION`/`_PKEY_VECTOR_SCORE` are emitted from an explicit
cursor over the DV-filtered local selection, advanced by each batch's actual
returned row count. The predicate-free guard is preserved (a row-filtering
predicate would desync the cursor), and the local-selection reader rejects
predicate readers self-consistently.
- Dropped the `first_row_id` requirement from the residual-filter position
collection; surviving rows' file-local positions are recovered from their
ordinal in the unfiltered scan.
The DE/append (global-index) vector path and the shared `SearchResult {
row_ids, scores }` shape are untouched — global row ids are legitimate there.
### Tests
- **Cross-language proof**: a real PK-vector table written by Java's
production `ivf-flat` indexer is committed as opaque binaries and read back via
`execute_read()` (`pk_vector_java_fixture_test.rs`). Its data files have no
`first_row_id` — exactly the condition that broke the old code. Provenance
(Java commit, generator command, schema/config, query + analytic top-k,
checksum) is recorded in the test header. This becomes a permanent regression
guard.
- `execute_read` with `first_row_id = None` selects the correct file-local
positions; with `first_row_id = Some(100)` (non-position-aligned) still returns
the correct rows — catches any accidental reuse of the global row-range
semantics.
- Position↔row alignment across a deletion vector and multiple batches.
- PK `execute_scored()` now returns an error; DE
`execute_scored`/`to_row_ranges` tests stay green.
`cargo test -p paimon` passes; `cargo clippy -p paimon --lib --tests -- -D
warnings` and `cargo fmt --check` clean.
### API and Format
No storage-format change. Behavior change: PK `execute_scored()`/`execute()`
now return a clear error instead of (incorrect) results computed under the
`first_row_id == 0` assumption — a corrective change on a path that was already
broken for real Java-written data. No public physical-coordinate result type is
added.
### Documentation
No documentation changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
