JingsongLi opened a new pull request, #8403: URL: https://github.com/apache/paimon/pull/8403
## Summary Support building Tantivy full-text global indexes from PyPaimon and align the local Python reader with the Java Tantivy query semantics where possible. ## Changes - Add a PyPaimon Tantivy full-text global index writer that produces Java-compatible archived Tantivy index files and sparse index option metadata. - Reuse the generic Python global-index build path for Tantivy full-text indexes, including row-range sharding and non-string column validation. - Align Python full-text query behavior with Java for match boost, fuzziness, boost/demote queries, boolean score composition, and explicit failures for unsupported Tantivy options. - Fail fast when Python build is asked to use `tantivy.tokenizer=jieba`, because tantivy-py does not expose the same Rust `tantivy_jieba` tokenizer used by Java. - Update full-text global-index docs with SQL/Python tabs for build/drop/query examples and document Python's `jieba` build limitation. ## Testing - [x] `python -m pytest -q paimon-python/pypaimon/tests/global_index_build_test.py paimon-python/pypaimon/tests/vector_search_filter_test.py` - [x] `python -m compileall -q paimon-python/pypaimon/globalindex/tantivy paimon-python/pypaimon/globalindex/create_global_index.py paimon-python/pypaimon/tests/global_index_build_test.py paimon-python/pypaimon/tests/vector_search_filter_test.py` - [x] `git diff --check` ## Notes PyPaimon can query Java/SQL-built `jieba` Tantivy indexes for match queries when the Python `jieba` package is installed, but Python index construction intentionally rejects `tantivy.tokenizer=jieba` to avoid silently producing a non-equivalent index. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
