shyjsarah opened a new pull request, #571: URL: https://github.com/apache/paimon-rust/pull/571
### Purpose Migrate the existing append/data-evolution global full-text search path from the legacy in-repository Tantivy implementation to the shared `paimon-ftindex-core` engine. This lets Rust and DataFusion read the current `full-text` index archives produced by Java Paimon and PyPaimon while preserving the existing `full_text_search(table_name, column_name, query_text, limit)` API. This PR intentionally focuses on the non-primary-key/global-index path. The primary-key full-text and hybrid search work is covered separately by #568. ### Brief change log - Replace the legacy Tantivy archive reader and writer modules with an adapter over `paimon-ftindex-core`. - Read `full-text` entries from snapshot index manifests through a bounded-concurrency `SeekRead` adapter. - Support plain-text queries, the Paimon JSON query DSL, Roaring include filters, shard-local/global row-ID translation, and global Top-K merging. - Align `fast`, `full`, and `detail` search modes with Java behavior, including raw fallback for uncovered row ranges. - Filter deleted rows through data-evolution deletion vectors and keep raw fallback scoring in a single BM25 corpus. - Add core and DataFusion coverage for native index reads, row-range filtering, raw fallback, query normalization, authorization, and branch table queries. ### Testing - [x] `cargo check -p paimon --features fulltext` - [x] `cargo test -p paimon --features fulltext full_text -- --nocapture` - [x] `cargo check -p paimon-datafusion --features fulltext` - [x] `cargo test -p paimon-datafusion --features fulltext fulltext_tests -- --nocapture` - [x] `cargo clippy -p paimon --lib --tests --features fulltext -- -D warnings` - [x] `cargo clippy -p paimon-datafusion --lib --tests --features fulltext -- -D warnings` ### API and format - No public API or table data-file format changes. - The SQL table-function signature remains unchanged. - The `fulltext` feature now reads the current Paimon `full-text` archive format. Compatibility with the legacy paimon-rust `tantivy-fulltext` archive is intentionally removed. ### Draft status and coordination `paimon-ftindex-core` is temporarily pinned to the public `v0.1.0-rc5` Git tag. The dependency will be changed to a crates.io version before this PR is marked ready, because a git-only dependency prevents publishing `paimon`. This work overlaps with #563 in dependency and reader wiring. It is being opened early so we can agree on the integration order and reuse or rebase on the reader foundation rather than duplicate it. It complements, but does not implement, the primary-key full-text and hybrid search scope in #568. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
