SteNicholas opened a new pull request, #559:
URL: https://github.com/apache/paimon-cpp/pull/559

   ### Purpose
   
   Linked issue: #406
   
   Full-text searches currently omit rows outside stored index coverage. Row 
filters can also discard matching rows when scalar index coverage is 
incomplete. Add `full` and `detail` modes to search these gaps from the planned 
snapshot, aligned with Java Paimon, while keeping `fast` as the default.
   
   - Resolve each family's search mode before falling back to 
`global-index.search-mode`. Compute unindexed ranges from the snapshot's next 
row ID in `full` mode and partition-filtered data manifests in `detail` mode, 
using Java's single-field coverage rule.
   - Read live, non-null text from raw ranges with partition filters and 
deletion vectors, then build temporary Tantivy or Lucene indexes entirely in 
memory. Use the existing backend formats without table-storage writes or a 
writable temporary directory.
   - Use the whole raw corpus for scoring, apply row filters before top-k, 
translate temporary document IDs to global row IDs and merge raw hits with 
stored-index results. Read gaps in scalar filter coverage in `full` and 
`detail` modes, with optional data refinement for inexact index candidates.
   - Make raw and stored search ranges disjoint before distributing splits. 
Preserve physical shard ranges for row ID translation and carry snapshot and 
schema IDs to keep worker reads bound to the planning data and fields.
   - Pass the macOS SDK and required libc++ compatibility flags to bundled 
Boost and Lucene++ builds, and correct reference access in Lucene++ container 
swaps.
   
   ### Tests
   
   Extend option-resolution, index-coverage and split-serialization unit tests 
for mode precedence, coverage gaps, raw split validation and legacy encodings. 
Share Tantivy and Lucene integration coverage for appended rows, partial 
partition coverage, row filters, deletion vectors, schema changes, distributed 
reads and operation without a writable temporary directory. Add a focused 
regression test for Lucene++ container swaps.
   
   ### API and Format
   
   - Add the exported `RawFullTextSearchSplit` interface and the 
`vector-index.search-mode` option, reserved for table-level vector search.
   - Extend the C++ full-text split encoding to version 3 with planning schema 
IDs, while continuing to read older encodings.
   - Keep persisted table and backend index file formats unchanged.
   
   ### Documentation
   
   Update the full-text search guide and public API comments to explain mode 
precedence, single-field index coverage, filter refinement, raw-corpus scoring, 
memory requirements and distributed execution.
   
   ### Generative AI tooling
   
   Generated-by: Codex (GPT-6)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to