JunRuiLee opened a new pull request, #599: URL: https://github.com/apache/paimon-rust/pull/599
### Purpose Part of #567 (primary-key full-text + hybrid search on the Rust read side), the third slice after #563 (the `paimon-ftindex-core` reader foundation) and #594 (the PK full-text data layer). This adds the primary-key full-text **read path**: plan splits over `"full-text"` index payloads, run bucket-local search through the `paimon-ftindex-core` reader with a live-row include filter, map archive row-ids to physical `(data file, row position)`, fuse cross-bucket by score, and materialize the matching rows best-score-first with a `__paimon_search_score` column. **FAST mode only**, mirroring Java (`FULL`/`DETAIL` fail loud; there is no exact/rerank fallback on the full-text path). ### Brief change log - **`PrimaryKeyFullTextScan` + `PrimaryKeyFullTextSearchSplit`**: pin a snapshot, scan the index manifest, group payloads by `(partition, bucket)` via the shared `PkFullTextBucketState` reconciliation (from #594), and emit splits. Split invariants mirror Java `PrimaryKeyFullTextSearchSplit` (unique data files; each current payload covers ≥1 active source; no double-covered source; covered ∪ uncovered == the active data files); buckets `< 0` are skipped. - **`PrimaryKeyFullTextBucketSearch`**: lay each payload's source files out as a cumulative archive (`offset_i` = running sum of prior source row counts); a source is active iff its data file is in the split; build a live-row include allow-list (union of active source ranges minus deletion-vector positions, each mapped to `offset + local`), or `None` when nothing is inactive/deleted; search the archive through the `paimon-ftindex-core` reader (query passed verbatim); map each returned archive row-id back to `row_position = row_id − source.offset`, failing loud on an inactive/out-of-range row-id or a source/data-file row-count mismatch. Archives are opened under the split's bucket directory (mirroring Java `IndexInDataFileDirPathFactory`). - **`PrimaryKeyFullTextCandidate` + `top_k_by_score`**: score-descending Top-K with a deterministic tie-break (score desc, then partition bytes, bucket, data file, row position), and finite-score validation (mirroring Java `PrimaryKeySearchPosition`). - **`PrimaryKeyFullTextRead`** (FAST-only): search every planned bucket, fuse by score into a global Top-`limit`, materialize the winning physical rows via the vector physical-position materializers, reorder them back to best-score order (keyed by full physical position), strip the internal position column, and append `__paimon_search_score`. Raw full-text scores are carried verbatim — no `distance_to_score` conversion. - **`FullTextSearchBuilder`** primary-key branch: `execute_read` materializes rows; `execute`/`execute_scored` fail loud on the PK path (physical positions, not global row-ids). Dispatch mirrors Java `primaryKeyFullTextDefinition` — the PK path is taken only when data evolution is disabled and the queried column is a configured `pk-full-text.index.columns` entry. The append/global-index path is unchanged. Reuses the primary-key vector physical-position materializers (`PkVectorIndexedSplit` / `PkVectorIndexedSplitRead` / `PkVectorPositionRead`), which are index-agnostic; the full-text read builds the indexed splits directly with raw scores rather than through the vector distance-to-score path. ### Tests - Scan/split: `(partition, bucket)` grouping, current/stale via the bucket state, uncovered files carried, split invariants, bucket `< 0` skip. - Bucket search: cumulative source-range offsets; `include = None` when clean; include subtracts deletion vectors (mapped by offset); row-id → position; fail-loud on inactive/out-of-range row-id and row-count mismatch; empty-include skip. - `top_k_by_score`: score-descending order + deterministic tie-break; cross-bucket fusion; non-finite score rejected. - FAST-only: `FULL`/`DETAIL` fail loud. - Builder: PK `execute_read` materializes rows best-score-first with the score column (round-trip against an archive built with the core writer, deletion-vector filtered); PK `execute`/`execute_scored` fail loud; append-path regression stays green. - `cargo test -p paimon` and `--features fulltext` green; `cargo clippy -p paimon --all-targets` (+ `--features fulltext`) `-D warnings` clean; `cargo fmt --all --check` clean; `cargo build -p paimon-datafusion` (Send boundary) passes. ### API and Format No storage-format change and no change to existing public read APIs. `FullTextSearchBuilder` gains `execute_read` for the primary-key full-text path. The archive-reading code is behind the existing `fulltext` cargo feature; scan/split/candidate/top-k are plain planning code. ### Documentation No documentation changes; module- and item-level doc comments describe the scan, bucket search, ranking, and read. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
