JunRuiLee opened a new pull request, #568: URL: https://github.com/apache/paimon-rust/pull/568
### Purpose Implements #567 — the **complete primary-key full-text and hybrid search read path** for the Rust side, mirroring Java Paimon. This is the full picture of what I've implemented locally (posted as one draft so the whole scope is visible); it builds on the reader foundation in #563. Depends on the shared native core `paimon-ftindex-core` (`apache/paimon-full-text`, tantivy 0.26.1) — the same engine the Java/Python bindings use — so Rust reads the exact on-disk archives Java writes. ### Scope (four parts, all implemented + tested locally) **1. Reader foundation** (= #563) - Optional dependency on `paimon-ftindex-core`, gated behind the `fulltext` feature. - `crates/paimon/src/ftindex/`: `FullTextArchiveReader` (`from_input_file` / `search` / `search_with_include` with a roaring live-row allow-list); fail-loud error mapping. **2. PK full-text data layer** - `CoreOptions::primary_key_full_text_index_columns()` (lenient, mirrors Java `pk-full-text.index.columns`). - Generalized the source-metadata leaf into a shared `spec/pk_index_source.rs` (`PrimaryKeyIndexSourceMeta` / `PrimaryKeyIndexSourceFile` + `should_read` policy), shared by the vector and full-text paths (mirrors Java `index/pk`). - `PkFullTextBucketState` — per-bucket current/stale payload reconciliation (`"full-text"` index type), mirroring Java `PkFullTextBucketIndexState`. **3. PK full-text read path (FAST mode)** - `PrimaryKeyFullTextScan` / `PrimaryKeyFullTextSearchSplit` (manifest scan, `(partition, bucket)` grouping, split invariants). - `PrimaryKeyFullTextBucketSearch` (cumulative `SourceRange` offsets, live-row `include` = active ranges − deletion vectors, archive row-id → physical position). - `PrimaryKeyFullTextRead` + a PK branch in `full_text_search_builder` — materializes rows by physical position + `__paimon_search_score`, best-score-first; `execute`/`execute_scored` fail loud on PK (physical positions, not global row ids). **4. Hybrid search on PK tables** - Shared `PrimaryKeySearchPosition` + `PrimaryKeySearchRanker` (RRF / weighted-score / weighted-MRR, `DEFAULT_RRF_K = 60`), a faithful port of Java `PrimaryKeySearchRanker`. - Hybrid builder PK path: fuse vector + full-text routes in physical-position space (one pinned snapshot across routes, per-route weights), then materialize — mirroring Java `HybridSearchBuilderImpl`. ### Tests Full `cargo test -p paimon --features fulltext` green (1921 passed / 0 failed), including the vector regression, PK-vector baseline, and the Java-written fixture. `cargo clippy -p paimon --lib --tests --features fulltext -- -D warnings` clean; default (no-feature) build and `cargo build -p paimon-datafusion` (Send boundary) both pass. ### API and Format No public API change to existing paths and no storage-format change; adds feature-gated readers over the Java-written archive formats (v1 full-text) via `paimon-ftindex-core`. > **Why draft:** `paimon-ftindex-core` is pinned to a **git** tag for now (not yet on crates.io); a git-only dependency blocks `cargo package`/publishing of `paimon`. Once the crate is released to crates.io, the dependency switches to a registry version and this is marked ready. I intend to split this into reviewable stacked PRs on top of #563 for merge; posting it whole here so the full scope is visible. ### Documentation Module-level doc comments describe each component. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
