JingsongLi opened a new pull request, #8463: URL: https://github.com/apache/paimon/pull/8463
## Summary Switch Paimon's Tantivy full-text global index implementation to the standalone `paimon-full-text` dependency for Java and PyPaimon. Single-column structured full-text DSL is delegated to the native reader, while Paimon keeps Java/Python orchestration for multi-column and hybrid query composition. ## Changes - Add the `paimon-full-text-index` Maven dependency and route Java full-text index reader/writer calls through `org.apache.paimon.index.fulltext`. - Replace PyPaimon's `tantivy-py` reader/writer path with `paimon_ftindex` adapters, including native roaring-filter support for `include_row_ids`. - Push whole single-column full-text queries down to the native reader; keep multi-match/cross-column planning in Paimon. - Remove index-module archive layout and searcher-pool helpers that were specific to `paimon-tantivy-jni`. - Update Java/Python tests and mixed-language E2E coverage for the new dependency. - Update CI workflows to checkout, build, and install `apache/paimon-full-text` for Java and Python jobs. ## Testing - [x] `python -m py_compile paimon-python/pypaimon/globalindex/tantivy/tantivy_full_text_global_index_reader.py paimon-python/pypaimon/globalindex/tantivy/tantivy_full_text_index_writer.py paimon-python/pypaimon/tests/vector_search_filter_test.py paimon-python/pypaimon/tests/global_index_build_test.py paimon-python/pypaimon/tests/e2e/java_py_read_write_test.py` - [x] `python -m pytest paimon-python/pypaimon/tests/vector_search_filter_test.py paimon-python/pypaimon/tests/global_index_build_test.py -q` - [x] `PAIMON_FTINDEX_JNI_LIB_PATH=... mvn -B -ntp -pl paimon-tantivy/paimon-tantivy-index -am -Dtest=TantivyFullTextGlobalIndexTest,JavaPyTantivyE2ETest,TantivyFullTextGlobalIndexerFactoryTest -Drun.e2e.tests=true -DfailIfNoTests=false -Dcheckstyle.skip=true -Dspotless.check.skip=true clean test` - [x] `PAIMON_FTINDEX_LIB_PATH=... PYTHONPATH=... python -m pytest java_py_read_write_test.py::JavaPyReadWriteTest::test_read_tantivy_full_text_index -q` - [x] `git diff --check --cached` ## Notes - Depends on apache/paimon-full-text#6 for Paimon query DSL aliases and logical-column compatibility in the native reader. - Python 3.6 keeps skipping the `paimon_ftindex` package installation because the package requires Python 3.8+. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
