HaHaJeff opened a new pull request, #224: URL: https://github.com/apache/paimon-cpp/pull/224
### Purpose Linked issue: #158 This PR extends the pluggable realtime read and write support introduced by #163 and the `RealtimeStore` API from #199 to fixed-bucket primary-key tables. Applications attach a `RealtimeContext` to the existing file-store paths. Its `RealtimeStoreFactory` creates an append or primary-key store from a typed request. The primary-key store keeps mutations in memory and passes sealed readers to `MergeTreeWriter` during prepare commit. For primary-key reads, immutable memory read views are combined with the selected disk snapshot through `RealtimeSplit`. Both sources enter the existing primary-key merge-on-read path, including sequence ordering, row-kind handling, and merge-engine semantics. Predicates that are unsafe before merge are evaluated after merge. The main changes are: - add an in-memory primary-key `RealtimeStore` implementation; - extend `RealtimeStoreFactory` with typed append and primary-key creation requests; - let `MergeTreeWriter` consume sealed readers produced by the primary-key store; - reuse the existing key-value reader and merge path for disk and memory data; - restore offsets from snapshot metadata and sequence numbers from committed bucket files; - reuse the realtime file-store and two-stage scan/read APIs introduced by #163 and #199; and - preserve non-realtime and append-realtime behavior. The built-in V1 primary-key implementation supports fixed-bucket tables with the deduplicate merge engine, full-row mutations, latest-snapshot recovery, concurrent readers, and internally synchronized write and prepare operations. Dynamic buckets, lookup or early MOR, aggregation and partial-update merge engines, data evolution, user sequence fields, explicit realtime-writer compaction, and recovery from a non-latest snapshot are not included. The built-in V1 primary-key store keeps realtime mutations entirely in memory and does not implement spill. The public `RealtimeStore` contract still permits custom implementations to use their own spill strategy. ### Tests Added unit coverage for: - typed factory creation for append and primary-key stores; - primary-key write validation, sealing, and foreign-handle rejection; - sequence ordering, row kinds, deduplication, and commit readers; - immutable read views and concurrent readers; - realtime offset progress and snapshot refresh; and - supported option validation and no-spill writer construction. Added integration coverage for: - primary-key realtime write, prepare, commit, refresh, and reopen; - latest-snapshot recovery of offsets and sequence numbers; - merge-on-read across committed files and memory segments; - deletes, repeated keys, projection, predicates, external compaction, and writer handoff; - concurrent write, prepare, commit, refresh, and read operations; and - non-realtime and append-realtime regression paths. The focused primary-key realtime tests pass under ASAN, UBSAN, and LeakSanitizer. ### API and Format This PR reuses the public realtime file-store APIs introduced by #163 and #199, including `RealtimeContext`, `RealtimeWriteBatch`, `PrepareCommitWithProgress`, `CommitWithProgress`, realtime split planning, and snapshot refresh. It does not add a separate primary-key table API. The factory API gains typed creation data: - `AppendRealtimeStoreCreateConfig` for append tables; - `PrimaryKeyRealtimeStoreCreateConfig` for primary-key tables; - `RealtimeStoreCreateConfig` as the request variant; and - `RealtimeStoreCreateRequest` for schema, options, memory pool, partition-bucket identity, and table-specific configuration. Custom factories implement `RealtimeStoreFactory::Create(RealtimeStoreCreateRequest&&)` and dispatch on the append or primary-key configuration. A primary-key implementation returns primary-key-sorted mutations with the row kind and sequence metadata required by the existing merge-on-read path. No new data-file or commit-message format is introduced. Primary-key realtime writes produce normal merge-tree data files and commit messages. Realtime offsets continue to use the versioned snapshot metadata introduced by #163; they are progress identifiers assigned by the framework and are not primary-key sequence numbers. Paimon serializes `Write` and `SealForCommit` for each store. Existing immutable read views remain valid across later writes, seals, refresh, and committed-offset reclamation. Realtime split tickets remain process-local and single-success-use as defined by #199. After a write or prepare failure, the caller discards the writer and context, recreates them from the latest committed snapshot, and replays its external WAL. Existing non-realtime tables and append-realtime tables retain their previous execution paths. ### Documentation The public headers document typed store creation, the primary-key reader contract, offset and sequence separation, supported V1 options, lifecycle conventions, and ownership of returned Arrow data. ### Generative AI tooling Generated-by: TraeCode (GPT-5) Reviewed-by: Claude Code (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
