SteNicholas opened a new pull request, #414: URL: https://github.com/apache/paimon-cpp/pull/414
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): <description> --> ### Purpose <!-- Linking this pull request to the issue --> Linked issue: #400 <!-- What is the purpose of the change --> Java Paimon writes and reads full-text indexes through the standalone [paimon-full-text-index](https://github.com/apache/paimon-full-text) 0.1.0 engine, while Paimon C++ used its own `crates/tantivy_ffi` (Tantivy 0.22) with a different archive layout, so neither side could read the other's index files. This PR replaces the in-tree Tantivy FFI and the `tantivy-fulltext` global index with a `full-text` global index (factory `full-text-global`) backed by the paimon-full-text-index 0.1.0 native engine (Tantivy 0.26.1), so Paimon C++ writes the same `PFTIDX01` index file format as Java Paimon. - **Indexer**: strips the `full-text.` prefix from the table options once, passes the remaining keys to the engine and stores them as flat JSON in `GlobalIndexIOMeta::metadata`, as Java `NativeFullTextIndexOptions#serialize` does. Readers use the analyzer configuration embedded in each index file. - **Writer** (aligned with Java `NativeFullTextGlobalIndexWriter`): accepts `STRING`/`CHAR`/`VARCHAR` fields, skips null values, writes no file for a shard without rows, and closes the output stream on every error path. Values and options that contain NUL characters are rejected, because the C API takes NUL-terminated strings. - **Reader** (aligned with Java `NativeFullTextGlobalIndexReader`): opens the index file lazily, serializes positional reads and allows concurrent searches. `FullTextSearch::query` is passed to the engine unchanged as a JSON DSL query, the pre-filter is pushed down as a portable 64-bit Roaring bitmap, and the top `limit` rows by BM25 score are returned. File system errors raised in the read or write callbacks are reported instead of the generic native error. - **Build**: the engine is built from its source release (overridable through `PAIMON_FULL_TEXT_URL`) with the shipped `Cargo.lock`, because a reader rejects index files written by another Tantivy version. CI requires Rust 1.90, and jieba is only built for `lucene-fts`. Remaining work for #400: cross-read fixtures written by Java Paimon itself. The checked-in fixtures are written by the engine's Python binding (`paimon-ftindex` 0.1.0), which wraps the same engine as Java Paimon. Aligning the `FullTextSearch` API is tracked in #401. ### Tests <!-- List UT and IT cases to verify this change --> UT, `src/paimon/global_index/full_text/full_text_global_index_test.cpp` (`paimon-full-text-index-test`): - `TestWriteAndSearch`, `TestWriterWithoutRows`, `TestWriterWithOnlyNullRows`, `TestRelativeRowIds` (including row ids beyond 32 bits), `TestPreFilterLimitAndMinScore` - `TestInvalidWrite`, `TestInvalidRead`, `TestInvalidIndexFile` (bad magic, truncated file, Tantivy version mismatch) - `TestConcurrentSearch` (shared pre-filter), `TestIOException`, `TestFlushFailureClosesOutputStream` - `TestCrossReadFixtures`: golden row ids and scores on fixtures written by `paimon-ftindex` 0.1.0 for the `default`, `ngram` and `jieba` tokenizers, and the same header metadata and results for archives written by Paimon C++. IT, `test/inte/global_index_test.cpp`: `TestFullTextWriteCommitScanReadIndex`, `TestWriteCommitScanReadFullTextIndexWithPartition`. CI: the `gcc-debug-x86_64` job runs `ci/scripts/build_paimon.sh --verify_full_text_archives`, which reads the archives written by Paimon C++ with the Python reader. These changes were not built or run locally; they rely on CI. ### API and Format <!-- Does this change affect API in include dir or storage format or protocol --> - No public API signature changes. Only the documentation of `FullTextSearch` in `include/paimon/predicate/full_text_search.h` is updated. - Storage format: the experimental `tantivy-fulltext` index is replaced by the `full-text` index, which uses the paimon-full-text-index archive format shared with Java Paimon. Existing `tantivy-fulltext` indexes must be rebuilt as `full-text` indexes. - Build option: `PAIMON_ENABLE_TANTIVY` is renamed to `PAIMON_ENABLE_FULL_TEXT`. ### Documentation <!-- Does this change introduce a new feature --> Yes. `docs/source/user_guide/global_index.rst` describes the `full-text` index, its queries, options and limitations, and `docs/source/building.rst` describes `PAIMON_ENABLE_FULL_TEXT`. ### Generative AI tooling Generated-by: Claude Code 2.1.286 (Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
