JunRuiLee opened a new pull request, #563:
URL: https://github.com/apache/paimon-rust/pull/563
### Purpose
First step of mirroring Java Paimon's primary-key full-text search into the
Rust read side. Java's full-text index is backed by a shared native Rust core,
`paimon-ftindex-core` (public repo `apache/paimon-full-text`, tantivy 0.26.1),
which the Java/Python bindings wrap via JNI/ctypes. This PR lets the Rust
reader depend on that **same** core directly (no JNI needed), so it can read
the on-disk archive format Java writes.
This is a deliberately small, self-contained foundation slice: dependency
wiring + a thin reader wrapper + round-trip tests. It de-risks the external
dependency (does it fetch and build here?) before any of the primary-key
scan/read machinery is built on top.
> Note: this track does not yet have a linked tracking issue (it is separate
from the PK-vector work). Opening as **draft**; issue link to be added.
### Brief change log
- Add optional git dependency `paimon-ftindex-core` (tag `v0.1.0-rc4`),
wired into the existing `fulltext` cargo feature via `dep:` syntax.
- New feature-gated module `crates/paimon/src/ftindex/` with
`FullTextArchiveReader`:
- `from_input_file(&InputFile)` — reads the archive bytes and opens the
core reader over an in-memory `SliceReader`.
- `search(query_json, limit)` — JSON-DSL query, returns `FullTextHits {
row_ids, scores }`.
- `search_with_include(query_json, limit, RoaringTreemap)` — restricts
results to a live-row allow-list (the `withIncludeRowIds` equivalent).
- Fail-loud error mapping (engine errors surface as `crate::Error`, no
silent fallback).
Out of scope (later PRs): shared PK-index-layer generalization,
`CoreOptions` full-text options, PK full-text scan/read, hybrid-on-PK, and
migrating the existing append/data-evolution full-text path off raw `tantivy
0.22` onto the core.
### Tests
- `ftindex::reader::tests::test_round_trip_search_returns_expected_rows` —
build an archive with the core writer, read it back, assert the correct
row-ids/scores.
- `ftindex::reader::tests::test_search_with_include_restricts_to_allow_list`
— all docs match; the roaring include-filter restricts results to the
allow-list subset (also validates cross-crate `roaring 0.11` compatibility).
- Full `cargo test -p paimon --features fulltext` green (lib 1724 passed / 0
failed); `cargo clippy -p paimon --lib --tests --features fulltext -- -D
warnings` clean; default (no-feature) build and `cargo build -p
paimon-datafusion` (Send boundary) both pass.
### API and Format
No public API change to existing paths and no storage-format change. Adds a
new feature-gated reader over the existing Java-written full-text archive
format (v1), read through the shared `paimon-ftindex-core` engine. The
`fulltext` feature transitionally pulls both tantivy 0.22 (legacy append path)
and 0.26 (via the core); the duplicate is removed when the append path
converges onto the core in a later PR.
### Documentation
No documentation changes; module-level doc comments describe the new reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
