Hi, all. If no any more comments. I'll start vote tomorrow Best regards, Yuxia
----- 原始邮件 ----- 发件人: "yuxia" <[email protected]> 收件人: "dev" <[email protected]> 发送时间: 星期六, 2026年 10 月 10日 下午 4:36:18 主题: Re: [DISCUSS] PIP-39: Introduce Lake-Stream Mode to Provide Second-Level Data Freshness for Paimon Tables Thanks Jingsong for the naming suggestion. It looks much better. I have updated the PIP accordingly. Best regards, Yuxia ----- 原始邮件 ----- 发件人: "Jingsong Li" <[email protected]> 收件人: "dev" <[email protected]> 发送时间: 星期六, 2026年 10 月 10日 下午 1:48:18 主题: Re: [DISCUSS] PIP-39: Introduce Lake-Stream Mode to Provide Second-Level Data Freshness for Paimon Tables Thanks Yuxia for your proposal! One minor comment: streamstore => lakestream.store? Best, Jingsong On Mon, Sep 28, 2026 at 7:14 PM yuxia <[email protected]> wrote: > > Hi, Xinyu. > > Thanks for sharing the details and current progress of PIP-46. > > From my understanding, PIP-39 and PIP-46 do not conflict and have little > direct overlap, because they address different layers and can be discussed > independently. > > PIP-39 defines the table-level read and write semantics for lake-stream mode. > The goal is to allow each engine to recognize when a Paimon table is > operating in lake-stream mode and consistently follow the behavior defined by > PIP-39. The Paimon connectors do not implement the Stream Store’s internal > read or write paths. Instead, they discover the configured Stream Store and > delegate the actual operations to its native source and sink implementations. > Therefore, PIP-39 defines the engine-facing integration contract and > observable table behavior, rather than the internal implementation of a > Stream Store. > > As I understand it, PIP-46 focuses on the implementation of native real-time > capabilities inside Paimon and paimon-cpp, including real-time state > management, split and commit message formats, offset tracking, and recovery. > These are implementation details of the real-time read and write path and are > outside the scope of PIP-39. > > A PIP-46 RealtimeStore implementation could be one way for an engine to > provide the real-time side of a lake-stream table. Similarly, a Fluss-backed > paimon-cpp RealtimeStore could be a useful integration to explore. However, I > think this can be discussed separately and should not be a prerequisite for > PIP-39. > > If a concrete integration later requires Paimon and a Stream Store to > exchange splits, offsets, or commit messages, we can discuss and align on a > common protocol at that point. For now, I believe PIP-39 and PIP-46 can > proceed independently. > > Best regards, > Yuxia > > ----- 原始邮件 ----- > 发件人: "刘欣瑀" <[email protected]> > 收件人: "dev" <[email protected]> > 发送时间: 星期一, 2026年 9 月 28日 下午 6:48:45 > 主题: Re:[DISCUSS] PIP-39: Introduce Lake-Stream Mode to Provide Second-Level > Data Freshness for Paimon Tables > > Hi Yuxia, > > Thank you for proposing PIP-39 and for the detailed design. We believe > improving data freshness is very important for Paimon, and the pluggable > Stream Store approach provides a promising way to combine Paimon’s historical > storage capabilities with a low-latency real-time data layer. > > Previously, PIP-46 proposed a similar real-time framework that provides > native, pluggable support for writing and querying real-time data inside > Paimon. The real-time state is process-local and recovery currently relies on > replaying data from the upstream system based on the durable offsets recorded > in Paimon snapshots. A pluggable RealtimeStore interface allows customized > storage implementations. > > The initial implementation of this framework has been completed in > paimon-cpp, and integration work with several query engines has already > started. > > Since PIP-39 and PIP-46 share several important concepts, we hope the two > designs can reuse a common protocol where possible. In particular, it would > be useful to align on: > - The representation and serialization of real-time splits and commit message > (may with realtime progress). Per-partition and per-bucket offsets, including > committed, tiered, and readable progress. > - The consistency boundary between a Paimon snapshot and the remaining > real-time data. > - Schema, row-kind, and primary-key merge semantics across the lake and > stream layers. > > We would also like to explore whether Fluss could provide a paimon-cpp > RealtimeStore plugin. Since paimon-cpp already provides some of the required > interfaces and real-time read/write capabilities, we believe starting the > integration with paimon-cpp could be a simpler and faster way to validate and > deliver this architecture. Such an integration could also help the two > proposal gradually align their data formats, split and offset protocols, and > lake-stream consistency semantics. This would allow the existing paimon-cpp > real-time framework to use Fluss as its durable real-time store, providing > stronger support for task restart and failover while keeping the integration > pluggable. We would be happy to collaborate on the common split and offset > protocol, as well as the Fluss-based RealtimeStore integration. > > Thanks again for the proposal. Looking forward to further discussion. > > Best regards, > Xinyu Liu > > At 2026-09-28 17:39:13, "yuxia" <[email protected]> wrote: > >Hi Paimon community, > > > >I would like to start a discussion on PIP-39: Introduce Lake-Stream Mode to > >Provide Second-Level Data Freshness for Paimon Tables. > > > >This PIP proposes a pluggable Stream Store integration for Paimon, with > >Fluss as the first implementation. Existing Paimon tables can enable > >lake-stream mode in place: Paimon continues to manage historical data, while > >the Stream Store provides second-level data freshness. > > > >The proposal defines the read and write semantics, the enablement and > >disablement procedures, and the corresponding Flink and Spark connector > >changes. > > > >Please find the proposal here: > >https://cwiki.apache.org/confluence/spaces/PAIMON/pages/398000424/PIP-39+Introduce+Lake-Stream+Mode+to+Provide+Second-Level+Data+Freshness+for+Paimon+Tables > > > >Feedback and suggestions are welcome. > > > >Best regards, > >Yuxia
