lxy-9602 opened a new pull request, #393: URL: https://github.com/apache/paimon-cpp/pull/393
<!-- PR titles must follow Conventional Commits: <type>(<optional-scope>): <description> --> ### Purpose <!-- Linking this pull request to the issue --> Linked issue: #158 Add a high-performance real-time deduplication mode for append tables. The mode keeps append writes efficient while using deletion vectors to hide replaced or deleted rows in real time. Data files remain ordinary append-table files, so all predicates and file indexes can still be used for pruning and pushdown across data files, sealed segments, and building data. Rows are deduplicated with last-write-wins semantics using a user-defined key, providing efficient real-time updates without introducing merge-tree read overhead. This change adds: - `realtime.deduplicate-key-fields`; - key-to-offset indexes for memory and data files; - deletion-vector generation for replaced or deleted rows; - consistent reads across data files, sealed segments, and building data; - committed snapshot refresh and memory reclamation. Current limitations: one non-null key field, fixed buckets, no data evolution or real-time spill, and `_REALTIME_OFFSET` requires a bitmap file index. <!-- What is the purpose of the change --> ### Tests Added unit and integration coverage for: - schema and option validation; - key/offset index lookup; - INSERT, DELETE, and UPDATE semantics; - deduplication across batches, segments, and committed files; - read-view stability, partition/bucket isolation, and predicate pushdown. <!-- List UT and IT cases to verify this change --> ### API and Format API impact: adds the deduplicate option and real-time store mode (`realtime.deduplicate-key-fields`). Format impact: deduplicate tables persist `_REALTIME_OFFSET`, key-to-offset sidecars, and deletion vectors. Existing table modes are unchanged. <!-- Does this change affect API in include dir or storage format or protocol --> ### Documentation <!-- Does this change introduce a new feature --> ### Generative AI tooling Generated-by: OpenAI Codex (GPT-5) <!-- If generative AI tooling has been used in the process of authoring this patch, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
