JingsongLi opened a new pull request, #4: URL: https://github.com/apache/paimon-full-text/pull/4
## Summary Adds first-class Chinese tokenization support by moving the Rust full-text stack to the current Tantivy/Jieba tokenizer line. The Rust core now registers `jieba` directly, so Java, Python, and FFI callers can enable it through the existing options path. ## Changes - Upgrade Tantivy to `0.26` and add `tantivy-jieba 0.20`, aligning both on `tantivy-tokenizer-api 0.7`. - Add `jieba.search-mode` and `jieba.ordinal-position` tokenizer options, enabled by default for recall and phrase-query correctness. - Wire `TokenizerKind::Jieba` into the existing `TextAnalyzer` pipeline with existing token filters. - Add coverage for Chinese term search, Chinese phrase queries, and prefixed Jieba option parsing. - Update README and integration/design docs to list `jieba` as a supported v1 tokenizer. ## Testing - [x] `cargo fmt --all --check` - [x] `git diff --check` - [x] `cargo test` - [x] `mvn -q -f java/pom.xml test` - [x] `PYTHONPATH=python python3 -m pytest -q python/tests` ## Notes - Raises the workspace Rust version from `1.75` to `1.86`, matching Tantivy `0.26.1`'s minimum supported Rust version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
