lxy-9602 opened a new pull request, #504: URL: https://github.com/apache/paimon-cpp/pull/504
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): <description> --> ### Purpose <!-- Linking this pull request to the issue --> Linked issue: #158 Add predicate indexes for realtime append-only and deduplicate reads. Queries can use indexed key and value fields to select candidate offsets before reading and filtering rows. - Introduce pluggable realtime index interfaces and per-field configuration through `realtime.index.<plugin>.columns`. - Provide an append-only hash index supporting equality and IN predicates, with concurrent readers sharing postings without copying index data. - Restrict index queries to `[disk snapshot committed end, pinned memory end)` and combine candidates with the pinned deletion vector. - Preserve candidate bitmaps through batch slicing and projection, skip non-matching batches, and evaluate residual predicates only on selected rows. - Keep memory indexes independent of persisted file indexes. File indexes are built from actual output rows after deduplication, preserving correct file row positions. Indexes remain in memory when segment data spills and are retained by pinned read views until released. <!-- What is the purpose of the change --> ### Tests Added or extended coverage for: - Hash postings, offset bounds, concurrent reads, plugin configuration, and conservative predicate fallback. - Store reads across segment rotation, spill, projection, bitmap slicing, and pinned DV snapshots. - Predicate evaluation on full, sparse, empty, and nullable selections. - Realtime integration scenarios covering index pruning without statistics or residual filtering, file-index row remapping after deduplication, snapshot boundaries, pinned plans across updates/deletes and refresh, and residual predicate composition. <!-- List UT and IT cases to verify this change --> ### API and Format - Add public `RealtimeIndex`, `RealtimeIndexReader`, and `RealtimeIndexFactory` interfaces. - Add a required `offset_range` to `RealtimeQueryContext`; callers must supply the bounds associated with the pinned memory view and selected disk snapshot. - No changes to persisted storage formats or protocols. <!-- Does this change affect API in include dir or storage format or protocol --> ### Documentation Introduce the optional configuration: `realtime.index.hash.columns=id,payload` <!-- Does this change introduce a new feature --> ### Generative AI tooling Generated-by: OpenAI Codex (GPT-6) <!-- If generative AI tooling has been used in the process of authoring this patch, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
