Hi Xinyu, Thanks for putting together this proposal. I’m excited about this feature—the ability to query newly written data before the next snapshot commit, while preserving Paimon’s existing append and primary-key semantics, could be very useful for low-latency workloads.
I have a few comments and questions about the current design. ### 1. Clarify the semantics and necessity of _OFFSET The proposal introduces a per-row _OFFSET system field and states that it is persisted and preserved during compaction. However, it is unclear whether _OFFSET becomes a table-level system column in Paimon, an internal physical field used only by the real-time path, or progress metadata maintained outside the table schema. This distinction affects schema compatibility, file readers, compaction, storage overhead, and support in Java and Rust clients, so it would be helpful to clarify it in the PIP. It is also unclear whether persisting _OFFSET for every row is necessary for union-read correctness. The design already requires commits to cover an ordered prefix of sealed segments. Could each segment instead carry its progress range, with Paimon persisting only the committed per-partition-bucket progress token in the snapshot? The real-time layer could then use that token to return uncommitted records after the snapshot boundary, without adding an extra physical field to every Paimon row. If a per-row _OFFSET is required, could the PIP explain which concrete requirement cannot be satisfied by segment-level and snapshot-level progress? Possible examples include partial-segment commits, user-visible ingestion ordering, row-level recovery, or exposing offsets through the reader API. It would also be useful to evaluate the corresponding storage, compaction, schema-evolution, and cross-client costs. ### 2. Clarify the intended query and deployment model The proposed API provides the same process-local RealtimeContext instance to both the write and scan contexts, while RealtimeSplit contains process-local MemIndexer and MemReadView references. Does this mean that real-time union reads are limited to applications that embed both the Paimon C++ writer and reader in the same process? >From the title, I initially understood that a regular application using the >Paimon C++ writer could enable this feature and make newly written data >queryable before the next snapshot commit, including from a separate Paimon >reader or query engine. However, my current understanding is that a writer-only application does not gain this capability. It must also embed the Paimon C++ scan/read path in the same process and expose its own query or serving interface if other users need access. Please clarify whether this understanding is correct, describe the intended deployment model, and explain who is expected to expose the query interface to users. ### 3. Consider a common abstraction for an optional real-time layer The capability proposed here appears more general than a C++-specific client feature. Real-time writes and disk/real-time union reads are useful table-level capabilities, but the current design is tightly coupled to a process-local C++ MemIndexer. This may create a fragmented user experience across the Paimon ecosystem. Java and Rust clients cannot use the same capability, and users may reasonably be confused about why real-time acceleration is available only when writing and querying through the C++ client. The C++-specific abstraction may also make future integrations with systems such as Fluss or Kafka-based real-time stores more difficult, because each integration may need to define its own write path, read path, and snapshot boundary. Would it make sense to consider a table-level abstraction for an optional real-time layer—something conceptually similar to a RealtimeStore, although the exact name and interface do not need to be decided in this PIP? Such an abstraction could be independent of programming language and implementation, and answer three basic questions: 1. How does Paimon determine whether a table has a real-time layer? 2. If a real-time layer exists, how should writers write data to it? 3. How should readers read from it and combine the result with committed Paimon data? Paimon could define common write and read protocols around this abstraction. The C++ MemIndexer could be the first implementation, while Java and Rust clients could implement the same protocols later. Systems such as Fluss or Kafka-based stores could potentially integrate as alternative implementations, allowing users to choose the option that best fits their deployment. Provider-specific details, such as lifecycle, durability, and deployment model, can remain within each implementation. The common abstraction only needs to provide a consistent way to discover, write to, and read from a real-time layer. I think starting with C++ is reasonable, but defining the common abstraction and semantics separately would provide a cleaner path for other clients and real-time providers in the future. Thanks again for the proposal. I’m looking forward to the discussion and the evolution of this feature. Best regards, Yuxia ----- 原始邮件 ----- 发件人: "周剑锋" <[email protected]> 收件人: "dev" <[email protected]> 发送时间: 星期三, 2026年 7 月 29日 下午 4:37:34 主题: Re:Re: [DISCUSS] PIP-46: Support pluggable real-time writes and memory/disk union reads for Paimon C++ Hi Xinyu, Thanks for the update. The added design addresses my concern. +1 for this. Best, Jianfeng On Wed, 29 Jul 2026 13:27:37 +0800 (CST), "刘欣瑀 " <[email protected]> wrote: > Hi Jianfeng, > > Thanks for the feedback. It is helpful to know that primary-key merge-on-read > is the higher-priority scenario for your workloads. This is aligned with the > proposal: memory data will be adapted into Paimon’s existing primary-key > merge pipeline rather than introducing separate merge semantics in MemIndexer. > > Your understanding of the commit and recovery model is correct. For the > initial implementation, we intend to allow only one in-flight PrepareCommit > and snapshot commit per real-time writer. PrepareCommit seals the current > building segment and immediately opens a new one, so ingestion can continue > while the sealed segment is being prepared. The committed _OFFSET progress is > published only after the corresponding snapshot commit succeeds. > > After recovery, the writer restores the per-partition-bucket _OFFSET progress > from the latest committed snapshot, initializes subsequent offsets from that > boundary, and relies on the external source to replay uncommitted input. > > If we later support preparing multiple sealed segments concurrently, snapshot > commits would still need to be serialized in segment order. For each > partition-bucket, committed progress could advance only through the ordered > prefix of committed segments; a later segment must not move _OFFSET past an > earlier uncommitted segment. To keep the initial correctness model > straightforward, concurrent in-flight commits are not included in the first > version. > > We will clarify this constraint in the proposal. > > Best, > Xinyu > > At 2026-07-28 19:42:35, "周剑锋" <[email protected]> wrote: > >Hi Xinyu, > > > >Thanks for the proposal. Overall, the proposed direction aligns well with > >our use cases. > > > >For our low-latency serving workloads, the higher-priority scenario is > >primary-key tables with merge-on-read semantics. Newly ingested records > >need to be queryable before the next snapshot commit and merged with > >committed disk data under Paimon’s existing primary-key semantics. We also > >have append-table use cases that require union reads across committed and > >real-time data. > > > >The proposed MemIndexer provides a suitable common framework, with append > >and primary-key semantics handled by their respective readers. > > > >I have one clarification about commit and recovery. Does the design assume > >only one in-flight PrepareCommit and snapshot commit per real-time writer? > >My understanding is that each PrepareCommit seals the current building > >segment and prepares the returned sealed segment. Snapshot progress > >advances only after the corresponding commit succeeds, and recovery then > >restores the next _OFFSET from the committed progress and replays > >uncommitted input. > > > >If multiple sealed segments can be prepared concurrently, are their commits > >serialized so that each partition-bucket’s committed _OFFSET advances only > >across a contiguous committed prefix? > > > >Best, > >Jianfeng > > > >On Tue, 28 Jul 2026 15:40:51 +0800 (CST), “刘欣瑀” [email protected] wrote: > > > >Hi everyone, > > > >I’d like to start a discussion on adding pluggable in-process real-time > >write and union-read support to Paimon C++. > >Problem > > > >Paimon uses committed snapshots as stable visibility boundaries. Records > >buffered by a writer cannot be queried through the standard reader until > >they are written to data files and committed into a snapshot. > > > >Some embedded and serving workloads require lower read latency: > > > >newly ingested records should be queryable before the next snapshot commit; > > > >ingestion should continue while PrepareCommit writes previously buffered > >records; > > > >queries should consistently combine committed disk data with uncommitted > >real-time data; > > > >append and primary-key tables should retain their existing Paimon semantics. > >Proposed Solution > > > >We propose an opt-in, process-local MemIndexer plugin integrated with the > >Paimon C++ write and read pipelines. > > > >The main ideas are: > > > >Paimon assigns an independent _OFFSET for each row within its > >partition-bucket. > > > >Each MemIndexer manages building and sealed segments. > > > >PrepareCommit seals the current segment and immediately opens a new one for > >continued ingestion. > > > >Sealed data is read from the plugin through commit readers and written by > >existing Paimon rolling writers. > > > >Per-partition-bucket _OFFSET progress is persisted through snapshot > >metadata. > > > >TableScan creates a RealtimeSplit containing disk splits and a pinned > >memory read view. > > > >Append tables concatenate disk and memory readers. > > > >Primary-key tables adapt memory readers into the existing Paimon merge > >pipeline. > > > >The plugin may use memory, local spill files, or specialized indexes > >internally, but Paimon continues to own table semantics, file formats, > >rolling policies, manifests, snapshots, and commit messages. > > > >The feature is disabled by default, so existing disk-only write and read > >paths are unaffected. > >Main Interfaces > > > >The proposal introduces several extension points: > > > >MemIndexer::Write > > > >MemIndexer::SealForCommit > > > >MemIndexer::AcquireReadView > > > >MemIndexer::CreateQueryReaders > > > >MemIndexer::CreateCommitReaders > > > >MemIndexer::Reclaim > > > >RealtimeContext > > > >internal RealtimeSplit planning > >PIP Document > > > >The full proposal, including lifecycle, public interfaces, append union > >read, and primary-key merge integration, is available here: > > > >https://cwiki.apache.org/confluence/spaces/PAIMON/pages/444334302/PIP-46+Support+pluggable+real-time+writes+and+memory+disk+union+reads+for+Paimon+C > >Looking for Feedback > > > >I’d appreciate community feedback on: > > > >The overall MemIndexer plugin boundary. > > > >The building/sealed segment lifecycle during PrepareCommit. > > > >The _OFFSET assignment and snapshot progress model. > > > >The integration with existing primary-key merge semantics. > > > >Any compatibility or implementation concerns. > > > >Looking forward to the discussion! > > > >Best regards, > > > >Xinyu
