lxy-9602 opened a new pull request, #546: URL: https://github.com/apache/paimon-cpp/pull/546
…-text search <!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): <description> --> ### Purpose <!-- Linking this pull request to the issue --> Linked issue: #158 Enable precise index evaluation for real-time append and deduplicate reads, combining scalar predicates with vector and full-text search. Scalar predicates and pinned deletion visibility are applied before TopK, so filtered or deleted rows do not consume result slots. Queries read each segment within the intersection of its pinned bounds and the requested offset range, excluding committed prefixes and later writes. - Reuse predicate evaluation, selectivity ordering, search filtering, and score completion between disk and real-time readers. - Introduce shared index reader capabilities and optional segment-level real-time index sources. - Support offset selection and deletion filtering when materializing real-time results. - Validate the pinned store schema before reading. Real-time primary-key merge remains unsupported in precise mode. Lumina real-time search integration is deferred to a separate PR; search behavior is covered with mocks. <!-- What is the purpose of the change --> ### Tests Added and updated unit tests for predicate evaluation, search capabilities, score alignment, offset bounds, deletion visibility, and pinned-source lifetime. Added six integration tests covering: 1. Exact scalar filtering before vector TopK, including unsupported and missing scalar indexes. 2. FullTextSearch and mixed FullTextMatch/scalar/vector predicates with separate score columns. 3. Reads across multiple disk files, sealed segments, and building segments. 4. In-memory deduplication updates and deletes before TopK. 5. Committed disk deletion vectors combined with real-time deletions across prepare, commit, and refresh. 6. Pinned plans and readers retaining their offset bounds and deletion visibility across writes and refresh. <!-- Describe tests added or updated and what they cover; explain if no test changes are needed. Omit execution commands, results, and reasons tests were not run locally. --> ### API and Format Public C++ API changes include: - `IndexReaderCapabilities`, shared by file and real-time index readers. - Real-time full-text/vector search methods and offset-based search result types. - Optional `RealtimeIndexedReadView` and `RealtimeIndexReadSource` interfaces. - Offset selection, offset deletions, and scalar-index pruning control in `RealtimeQueryContext`. The reader inheritance and query-context layout change. No storage format or serialized protocol changes. <!-- Does this change affect API in include dir or storage format or protocol --> ### Documentation Updated API comments for supported table modes, row-ID domains, pinned visibility, and search filtering requirements. <!-- Does this change introduce a new feature --> ### Generative AI tooling Generated-by: OpenAI Codex <!-- If generative AI tooling has been used in the process of authoring this patch, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
