JunRuiLee opened a new pull request, #594:
URL: https://github.com/apache/paimon-rust/pull/594
### Purpose
Part of #567 (primary-key full-text + hybrid search on the Rust read side),
the second slice after #563 (the `paimon-ftindex-core` reader foundation).
This adds the read-side **data layer** for primary-key full-text search —
everything the scan/read path needs *before* opening and searching an archive,
all independently unit-testable:
- the config that identifies a PK full-text column,
- the shared primary-key-index source-metadata leaf, generalized from the
existing PK-vector code to mirror Java's shared `index/pk/` package,
- the per-bucket current/stale payload reconciliation, mirroring Java
`PkFullTextBucketIndexState.fromActiveDataFiles`.
No archive I/O here, so nothing new is feature-gated — this is plain
data-layer code, like the existing PK-vector data layer.
### Brief change log
- **CoreOptions:** `primary_key_full_text_index_enabled()` /
`primary_key_full_text_index_columns()`, mirroring Java
`pk-full-text.index.columns` — no default; `split(',', -1)` then trim with
blank tokens preserved; returns a plain `Vec<String>` (never errors); `[]` only
when the key is absent.
- **Shared PK-index source-meta leaf:** relocate `spec/pk_vector_source.rs`
→ `spec/pk_index_source.rs`, rename `PkVectorSourceMeta`/`PkVectorSourceFile` →
`PrimaryKeyIndexSourceMeta`/`PrimaryKeyIndexSourceFile`, and move the shared
source policy (`should_read_pk_index_source`) there — mirroring Java's shared
`index/pk/` layer used by both vector and full-text. Byte format unchanged; all
PK-vector call sites re-pointed; behavior-preserving.
- **Full-text index type + bucket state:** `PK_FULL_TEXT_INDEX_TYPE =
"full-text"` and `PkFullTextBucketState::from_active_data_files(...)` — the
independent current/stale reconciliation mirroring Java
`PkFullTextBucketIndexState`, reusing the shared leaf. Per-payload defects
(row-count / row-range / overflow / source-set mismatch / field mismatch with
source-meta / parse failure) are collected as stale; only cross-payload
conflicts in the surviving current set (duplicate payload name, double-covered
source) fail loud; wrong index type / missing global-index-meta / field
mismatch without source-meta are ignored. The source-file → current-payload map
preserves insertion order (matching Java's `LinkedHashMap`) so the read path
can build deterministically ordered splits.
Note the public rename of the two source-meta spec types with no deprecated
alias: the crate is pre-1.0, these are niche read-internal types, and the
rename aligns with Java's shared `index/pk` naming.
### Tests
- CoreOptions: absent key → disabled + `[]`; `"a, b ,c"` → `["a","b","c"]`;
blank-token preservation (`"a,,b"` → `["a","","b"]`, `" , "` → `["",""]`);
single column.
- Shared leaf: the existing source-meta serde round-trip tests move over
unchanged under the new names; `should_read` truth table (COMPACT + level>0
true; APPEND / level 0 / absent false).
- Bucket state — one test per reconciliation outcome (15 cases): exact match
→ current; source drift / reorder-only / missing level / field mismatch with
source-meta / corrupt bytes / row-count / row-range / overflow → stale; field
mismatch without source-meta → ignored; wrong index type / no global-index-meta
→ ignored; two payloads at one level → both stale; duplicate payload name /
double-covered source → error.
- PK-vector regression suite (including the committed Java PK-vector
fixture) stays green after the leaf rename.
- `cargo test -p paimon` (1832) and `--features fulltext` (1853) green;
`cargo clippy -p paimon --all-targets` (+ `--features fulltext`) `-D warnings`
clean; `cargo fmt --all --check` clean; `cargo build -p paimon-datafusion`
(Send boundary) passes.
### API and Format
Public rename of the two PK-index source-meta spec types
(`PkVectorSourceMeta`/`PkVectorSourceFile` →
`PrimaryKeyIndexSourceMeta`/`PrimaryKeyIndexSourceFile`), no alias — pre-1.0,
read-internal. The on-disk source-metadata byte format is unchanged; no
storage-format change. The new CoreOptions accessors are additive.
### Documentation
No documentation changes; module- and item-level doc comments describe the
new options, the shared leaf, and the reconciliation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
