lxy-9602 opened a new pull request, #504:
URL: https://github.com/apache/paimon-cpp/pull/504

   <!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): 
<description> -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: #158 
   
   Add predicate indexes for realtime append-only and deduplicate reads. 
Queries can use indexed key and value fields to select candidate offsets before 
reading and filtering rows.
   
   - Introduce pluggable realtime index interfaces and per-field configuration 
through `realtime.index.<plugin>.columns`.
   - Provide an append-only hash index supporting equality and IN predicates, 
with concurrent readers sharing postings without copying index data.
   - Restrict index queries to `[disk snapshot committed end, pinned memory 
end)` and combine candidates with the pinned deletion vector.
   - Preserve candidate bitmaps through batch slicing and projection, skip 
non-matching batches, and evaluate residual predicates only on selected rows.
   - Keep memory indexes independent of persisted file indexes. File indexes 
are built from actual output rows after deduplication, preserving correct file 
row positions.
   
   Indexes remain in memory when segment data spills and are retained by pinned 
read views until released.
   
   <!-- What is the purpose of the change -->
   
   ### Tests
   
   Added or extended coverage for:
   
   - Hash postings, offset bounds, concurrent reads, plugin configuration, and 
conservative predicate fallback.
   - Store reads across segment rotation, spill, projection, bitmap slicing, 
and pinned DV snapshots.
   - Predicate evaluation on full, sparse, empty, and nullable selections.
   - Realtime integration scenarios covering index pruning without statistics 
or residual filtering, file-index row remapping after deduplication, snapshot 
boundaries, pinned plans across updates/deletes and refresh, and residual 
predicate composition.
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   - Add public `RealtimeIndex`, `RealtimeIndexReader`, and 
`RealtimeIndexFactory` interfaces.
   - Add a required `offset_range` to `RealtimeQueryContext`; callers must 
supply the bounds associated with the pinned memory view and selected disk 
snapshot.
   - No changes to persisted storage formats or protocols.
   
   <!-- Does this change affect API in include dir or storage format or 
protocol -->
   
   ### Documentation
   
   Introduce the optional configuration:
   
   `realtime.index.hash.columns=id,payload`
   
   <!-- Does this change introduce a new feature -->
   
   ### Generative AI tooling
   Generated-by: OpenAI Codex (GPT-6)
   <!--
   If generative AI tooling has been used in the process of authoring this 
patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling 
Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to