Hi Yuxia, Thanks for the detailed feedback. These questions helped us clarify the scope and simplify the design.
1. _OFFSET The original per-row _OFFSET was introduced for several engine-integration scenarios. After further discussion, we agree that it is unnecessary for the common real-time framework. We plan to remove the physical _OFFSET. Progress will be maintained through contiguous memory-segment ranges, while committed per-partition-bucket progress is persisted in snapshot properties. This avoids changes to table schemas, data files, compaction, and other language SDKs. 2. Query and deployment model Your understanding is correct: the current direct union-read implementation is process-local. The writer and reader share the same RealtimeContext, and the hosting application or serving engine exposes the query interface. A writer-only application does not make uncommitted data visible to an independent Paimon reader. As a future extension, the real-time portion can also be returned independently. An external engine could read a committed Paimon snapshot, obtain its progress boundary, fetch newer real-time records, and perform append union or primary-key merge itself. 3. Common real-time abstraction We agree that this can evolve into a common RealtimeStore abstraction. The current API discussion and draft implementation are available here: https://github.com/apache/paimon-cpp/issues/158 In the current draft: The host application currently enables the real-time layer by providing a RealtimeContext. It is not yet discovered through table metadata; a table option may be added in the future to identify and configure the real-time provider. Writers send records to the configured MemIndexer. PrepareCommitWithProgress returns commit messages with progress ranges, and CommitWithProgress commits them while persisting partition-bucket progress in the snapshot. Readers use the snapshot progress as the disk/real-time boundary. Paimon C++ can perform the union read internally, and a future interface may return the real-time result separately for an external engine to merge. A future table-level abstraction could add provider discovery and allow Java, Rust, Fluss, Kafka-based stores, or other implementations to follow the same write, commit, and read-progress contract. We would be very happy to collaborate with you, and contributions to the Paimon C++ real-time support are very welcome. Best regards, Xinyu At 2026-08-05 15:16:56, "yuxia" <[email protected]> wrote: >Hi Xinyu, > >Thanks for putting together this proposal. I’m excited about this feature—the >ability to query newly written data before the next snapshot commit, while >preserving Paimon’s existing append and primary-key semantics, could be very >useful for low-latency workloads. > >I have a few comments and questions about the current design. > >### 1. Clarify the semantics and necessity of _OFFSET > >The proposal introduces a per-row _OFFSET system field and states that it is >persisted and preserved during compaction. However, it is unclear whether >_OFFSET becomes a table-level system column in Paimon, an internal physical >field used only by the real-time path, or progress metadata maintained outside >the table schema. > >This distinction affects schema compatibility, file readers, compaction, >storage overhead, and support in Java and Rust clients, so it would be helpful >to clarify it in the PIP. > >It is also unclear whether persisting _OFFSET for every row is necessary for >union-read correctness. The design already requires commits to cover an >ordered prefix of sealed segments. Could each segment instead carry its >progress range, with Paimon persisting only the committed per-partition-bucket >progress token in the snapshot? The real-time layer could then use that token >to return uncommitted records after the snapshot boundary, without adding an >extra physical field to every Paimon row. > >If a per-row _OFFSET is required, could the PIP explain which concrete >requirement cannot be satisfied by segment-level and snapshot-level progress? >Possible examples include partial-segment commits, user-visible ingestion >ordering, row-level recovery, or exposing offsets through the reader API. It >would also be useful to evaluate the corresponding storage, compaction, >schema-evolution, and cross-client costs. > >### 2. Clarify the intended query and deployment model > >The proposed API provides the same process-local RealtimeContext instance to >both the write and scan contexts, while RealtimeSplit contains process-local >MemIndexer and MemReadView references. > >Does this mean that real-time union reads are limited to applications that >embed both the Paimon C++ writer and reader in the same process? > >From the title, I initially understood that a regular application using the >Paimon C++ writer could enable this feature and make newly written data >queryable before the next snapshot commit, including from a separate Paimon >reader or query engine. > >However, my current understanding is that a writer-only application does not >gain this capability. It must also embed the Paimon C++ scan/read path in the >same process and expose its own query or serving interface if other users need >access. > >Please clarify whether this understanding is correct, describe the intended >deployment model, and explain who is expected to expose the query interface to >users. > >### 3. Consider a common abstraction for an optional real-time layer > >The capability proposed here appears more general than a C++-specific client >feature. Real-time writes and disk/real-time union reads are useful >table-level capabilities, but the current design is tightly coupled to a >process-local C++ MemIndexer. > >This may create a fragmented user experience across the Paimon ecosystem. Java >and Rust clients cannot use the same capability, and users may reasonably be >confused about why real-time acceleration is available only when writing and >querying through the C++ client. > >The C++-specific abstraction may also make future integrations with systems >such as Fluss or Kafka-based real-time stores more difficult, because each >integration may need to define its own write path, read path, and snapshot >boundary. > >Would it make sense to consider a table-level abstraction for an optional >real-time layer—something conceptually similar to a RealtimeStore, although >the exact name and interface do not need to be decided in this PIP? > >Such an abstraction could be independent of programming language and >implementation, and answer three basic questions: > >1. How does Paimon determine whether a table has a real-time layer? >2. If a real-time layer exists, how should writers write data to it? >3. How should readers read from it and combine the result with committed >Paimon data? > >Paimon could define common write and read protocols around this abstraction. >The C++ MemIndexer could be the first implementation, while Java and Rust >clients could implement the same protocols later. Systems such as Fluss or >Kafka-based stores could potentially integrate as alternative implementations, >allowing users to choose the option that best fits their deployment. > >Provider-specific details, such as lifecycle, durability, and deployment >model, can remain within each implementation. The common abstraction only >needs to provide a consistent way to discover, write to, and read from a >real-time layer. > >I think starting with C++ is reasonable, but defining the common abstraction >and semantics separately would provide a cleaner path for other clients and >real-time providers in the future. > >Thanks again for the proposal. I’m looking forward to the discussion and the >evolution of this feature. > >Best regards, >Yuxia > >----- 原始邮件 ----- >发件人: "周剑锋" <[email protected]> >收件人: "dev" <[email protected]> >发送时间: 星期三, 2026年 7 月 29日 下午 4:37:34 >主题: Re:Re: [DISCUSS] PIP-46: Support pluggable real-time writes and >memory/disk union reads for Paimon C++ > >Hi Xinyu, > >Thanks for the update. The added design addresses my concern. > >+1 for this. > >Best, >Jianfeng > >On Wed, 29 Jul 2026 13:27:37 +0800 (CST), "刘欣瑀 " <[email protected]> wrote: >> Hi Jianfeng, >> >> Thanks for the feedback. It is helpful to know that primary-key >> merge-on-read is the higher-priority scenario for your workloads. This is >> aligned with the proposal: memory data will be adapted into Paimon’s >> existing primary-key merge pipeline rather than introducing separate merge >> semantics in MemIndexer. >> >> Your understanding of the commit and recovery model is correct. For the >> initial implementation, we intend to allow only one in-flight PrepareCommit >> and snapshot commit per real-time writer. PrepareCommit seals the current >> building segment and immediately opens a new one, so ingestion can continue >> while the sealed segment is being prepared. The committed _OFFSET progress >> is published only after the corresponding snapshot commit succeeds. >> >> After recovery, the writer restores the per-partition-bucket _OFFSET >> progress from the latest committed snapshot, initializes subsequent offsets >> from that boundary, and relies on the external source to replay uncommitted >> input. >> >> If we later support preparing multiple sealed segments concurrently, >> snapshot commits would still need to be serialized in segment order. For >> each partition-bucket, committed progress could advance only through the >> ordered prefix of committed segments; a later segment must not move _OFFSET >> past an earlier uncommitted segment. To keep the initial correctness model >> straightforward, concurrent in-flight commits are not included in the first >> version. >> >> We will clarify this constraint in the proposal. >> >> Best, >> Xinyu >> >> At 2026-07-28 19:42:35, "周剑锋" <[email protected]> wrote: >> >Hi Xinyu, >> > >> >Thanks for the proposal. Overall, the proposed direction aligns well with >> >our use cases. >> > >> >For our low-latency serving workloads, the higher-priority scenario is >> >primary-key tables with merge-on-read semantics. Newly ingested records >> >need to be queryable before the next snapshot commit and merged with >> >committed disk data under Paimon’s existing primary-key semantics. We also >> >have append-table use cases that require union reads across committed and >> >real-time data. >> > >> >The proposed MemIndexer provides a suitable common framework, with append >> >and primary-key semantics handled by their respective readers. >> > >> >I have one clarification about commit and recovery. Does the design assume >> >only one in-flight PrepareCommit and snapshot commit per real-time writer? >> >My understanding is that each PrepareCommit seals the current building >> >segment and prepares the returned sealed segment. Snapshot progress >> >advances only after the corresponding commit succeeds, and recovery then >> >restores the next _OFFSET from the committed progress and replays >> >uncommitted input. >> > >> >If multiple sealed segments can be prepared concurrently, are their commits >> >serialized so that each partition-bucket’s committed _OFFSET advances only >> >across a contiguous committed prefix? >> > >> >Best, >> >Jianfeng >> > >> >On Tue, 28 Jul 2026 15:40:51 +0800 (CST), “刘欣瑀” [email protected] wrote: >> > >> >Hi everyone, >> > >> >I’d like to start a discussion on adding pluggable in-process real-time >> >write and union-read support to Paimon C++. >> >Problem >> > >> >Paimon uses committed snapshots as stable visibility boundaries. Records >> >buffered by a writer cannot be queried through the standard reader until >> >they are written to data files and committed into a snapshot. >> > >> >Some embedded and serving workloads require lower read latency: >> > >> >newly ingested records should be queryable before the next snapshot commit; >> > >> >ingestion should continue while PrepareCommit writes previously buffered >> >records; >> > >> >queries should consistently combine committed disk data with uncommitted >> >real-time data; >> > >> >append and primary-key tables should retain their existing Paimon semantics. >> >Proposed Solution >> > >> >We propose an opt-in, process-local MemIndexer plugin integrated with the >> >Paimon C++ write and read pipelines. >> > >> >The main ideas are: >> > >> >Paimon assigns an independent _OFFSET for each row within its >> >partition-bucket. >> > >> >Each MemIndexer manages building and sealed segments. >> > >> >PrepareCommit seals the current segment and immediately opens a new one for >> >continued ingestion. >> > >> >Sealed data is read from the plugin through commit readers and written by >> >existing Paimon rolling writers. >> > >> >Per-partition-bucket _OFFSET progress is persisted through snapshot >> >metadata. >> > >> >TableScan creates a RealtimeSplit containing disk splits and a pinned >> >memory read view. >> > >> >Append tables concatenate disk and memory readers. >> > >> >Primary-key tables adapt memory readers into the existing Paimon merge >> >pipeline. >> > >> >The plugin may use memory, local spill files, or specialized indexes >> >internally, but Paimon continues to own table semantics, file formats, >> >rolling policies, manifests, snapshots, and commit messages. >> > >> >The feature is disabled by default, so existing disk-only write and read >> >paths are unaffected. >> >Main Interfaces >> > >> >The proposal introduces several extension points: >> > >> >MemIndexer::Write >> > >> >MemIndexer::SealForCommit >> > >> >MemIndexer::AcquireReadView >> > >> >MemIndexer::CreateQueryReaders >> > >> >MemIndexer::CreateCommitReaders >> > >> >MemIndexer::Reclaim >> > >> >RealtimeContext >> > >> >internal RealtimeSplit planning >> >PIP Document >> > >> >The full proposal, including lifecycle, public interfaces, append union >> >read, and primary-key merge integration, is available here: >> > >> >https://cwiki.apache.org/confluence/spaces/PAIMON/pages/444334302/PIP-46+Support+pluggable+real-time+writes+and+memory+disk+union+reads+for+Paimon+C >> >Looking for Feedback >> > >> >I’d appreciate community feedback on: >> > >> >The overall MemIndexer plugin boundary. >> > >> >The building/sealed segment lifecycle during PrepareCommit. >> > >> >The _OFFSET assignment and snapshot progress model. >> > >> >The integration with existing primary-key merge semantics. >> > >> >Any compatibility or implementation concerns. >> > >> >Looking forward to the discussion! >> > >> >Best regards, >> > >> >Xinyu
