Hi devs, I would like to share the follow-up details of this proposal. We have implemented the solution proposed by Yong Fang in PIP-30 on our internal version of Paimon. As we encountered several issues during implementation, the final implementation differs from the original PIP-30 document. I have prepared an updated version of the document, with all changes highlighted using a yellow background. The corresponding figures below the relevant text have also been revised accordingly.
https://docs.google.com/document/d/1asWKzoytfeB1D8bS_yRIAHnpR40frLf0fnPn2-WSL74/edit?usp=sharing We look forward to your feedback and comments. Thanks, Biao /'bɪ.aʊ/ On Wed, 19 Feb 2025 at 10:23, Yong Fang <[email protected]> wrote: > Thanks for all the feedback, if there are no more questions, I will > initiate a vote later. > > Best, > Fang Yong > > On Thu, Feb 13, 2025 at 12:52 PM xiangyu feng <[email protected]> > wrote: > > > Hi Yong, > > > > I see your point now. The overall design LGTM. > > > > Besides the support for region failover, we can also consider how to > > support regional committer in the future. Each region can advance its > data > > commitment mutually independent. This will also bring great value. > > > > Best, > > Xiangyu > > > > Jingsong Li <[email protected]> 于2025年2月12日周三 22:25写道： > > > > > Thanks Yong for the explanation. > > > > > > Sounds good to me. > > > > > > Best, > > > Jingsong > > > > > > On Wed, Feb 12, 2025 at 6:06 PM Yunfeng Zhou > > > <[email protected]> wrote: > > > > > > > > Hi Yong, > > > > > > > > > It is necessary to create a Paimon > > > > snapshot, and compaction may even be triggered. Therefore, > asynchronous > > > > threads should be considered for these operations to avoid blocking > the > > > > main thread of the JM. > > > > > > > > This statement sounds like it would be JM to perform the compaction, > > > regardless of sync or async. AFAIK, it should be writer operators to do > > the > > > compact, so JM need not consider this kind of overhead. > > > > > > > > Just a correction on a possible misunderstanding. The design and > > > response itself sounds good to me. > > > > > > > > Best, > > > > Yunfeng > > > > > > > > > 2025年2月11日 16:55，Yong Fang <[email protected]> 写道： > > > > > > > > > > It is necessary to create a Paimon > > > > > snapshot, and compaction may even be triggered. Therefore, > > asynchronous > > > > > threads should be considered for these operations to avoid blocking > > the > > > > > main thread of the JM. > > > > > > > > > >
