JingsongLi opened a new pull request, #8648: URL: https://github.com/apache/paimon/pull/8648
## Summary Add compaction-visible full-text indexing and search for primary-key tables. The index follows the primary-key table lifecycle, filters physical rows with deletion vectors, and incrementally consolidates immutable archives with the shared primary-key index LSM policy. ## Changes - Add primary-key full-text configuration, schema validation, deterministic analyzer fingerprints, and source metadata for ordered multi-file archives. - Build bucket-local native full-text archives from level-1-or-higher compact output and maintain them asynchronously with shared fanout and stale-ratio compaction options. - Support multi-source archive row-ID mapping, deletion-vector offsets, atomic publish/rollback, stale-source reclamation, and compatibility with existing single-source archives. - Add snapshot-scoped full-text scan/read/search over physical data-file positions, deterministic RRF ranking, temporary-index fallback for uncovered eligible files, and hybrid vector/full-text fusion. - Expose primary-key full-text search through Spark SQL and the Flink `full_text_search` procedure, with documentation for configuration, visibility, maintenance, and limitations. ## Testing - [x] JDK 8: 21 focused `paimon-core` test classes (144 tests) - [x] Spark 3 / JDK 8: `FullTextSearchTest` and `HybridSearchTest` (13 tests) - [x] Spark 4 / JDK 17: `FullTextSearchTest` and `HybridSearchTest` (13 tests) - [x] Flink 1 / JDK 8: `FullTextSearchProcedureITCase` - [x] Flink 2 / JDK 11: `FullTextSearchProcedureITCase` - [x] JDK 8: clean `paimon-core` and `paimon-full-text` compile with Checkstyle, Spotless, and Maven Enforcer - [x] Native primary-key full-text contract test invoked; it runs when `PAIMON_FTINDEX_JNI_LIB_PATH` is available ## Notes - Primary-key full-text uses the fixed `full-text` implementation; no `index.type` option is introduced. - Level-0 foreground files remain outside full-text coverage until compaction. - The first release supports one character full-text column per primary-key table. - Python API changes are intentionally out of scope. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
