SteNicholas opened a new pull request, #559: URL: https://github.com/apache/paimon-cpp/pull/559
### Purpose Linked issue: #406 Full-text searches currently omit rows outside stored index coverage. Row filters can also discard matching rows when scalar index coverage is incomplete. Add `full` and `detail` modes to search these gaps from the planned snapshot, aligned with Java Paimon, while keeping `fast` as the default. - Resolve each family's search mode before falling back to `global-index.search-mode`. Compute unindexed ranges from the snapshot's next row ID in `full` mode and partition-filtered data manifests in `detail` mode, using Java's single-field coverage rule. - Read live, non-null text from raw ranges with partition filters and deletion vectors, then build temporary Tantivy or Lucene indexes entirely in memory. Use the existing backend formats without table-storage writes or a writable temporary directory. - Use the whole raw corpus for scoring, apply row filters before top-k, translate temporary document IDs to global row IDs and merge raw hits with stored-index results. Read gaps in scalar filter coverage in `full` and `detail` modes, with optional data refinement for inexact index candidates. - Make raw and stored search ranges disjoint before distributing splits. Preserve physical shard ranges for row ID translation and carry snapshot and schema IDs to keep worker reads bound to the planning data and fields. - Pass the macOS SDK and required libc++ compatibility flags to bundled Boost and Lucene++ builds, and correct reference access in Lucene++ container swaps. ### Tests Extend option-resolution, index-coverage and split-serialization unit tests for mode precedence, coverage gaps, raw split validation and legacy encodings. Share Tantivy and Lucene integration coverage for appended rows, partial partition coverage, row filters, deletion vectors, schema changes, distributed reads and operation without a writable temporary directory. Add a focused regression test for Lucene++ container swaps. ### API and Format - Add the exported `RawFullTextSearchSplit` interface and the `vector-index.search-mode` option, reserved for table-level vector search. - Extend the C++ full-text split encoding to version 3 with planning schema IDs, while continuing to read older encodings. - Keep persisted table and backend index file formats unchanged. ### Documentation Update the full-text search guide and public API comments to explain mode precedence, single-field index coverage, filter refinement, raw-corpus scoring, memory requirements and distributed execution. ### Generative AI tooling Generated-by: Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
