Hi Yong, thanks for driving this PIP. I have a small question: Why not use Flink SinkV2 instead of moving the committer logic into JM's OperatorCoordinator?
On Tue, Feb 11, 2025 at 10:34 AM Jingsong Li <jingsongl...@gmail.com> wrote: > > Thanks Yong for driving this PIP! > > >> Currently, there will be a global committer node in Flink Paimon job which > >> is used to ensure the consistency of written data in Paimon. This > >> committer node will connect all the tasks in Flink job together, and all > >> the tasks are within one region. As a result, when any task fails, it will > >> trigger a global failover of the Flink job. We use HDFS as the remote > >> storage, and we often encounter situations where the global failover of > >> jobs is triggered due to write timeouts or errors when writing to HDFS, > >> which are quite a few stability issues. > > I know that Flink SinkV2 is also committed through a regular node, > does this mean that SinkV2 also has this drawback? > > Best, > Jingsong > > On Mon, Feb 10, 2025 at 4:06 PM Yunfeng Zhou > <flink.zhouyunf...@gmail.com> wrote: > > > > Hi Yong, > > > > The general idea looks good to me. Is there any statistics on the number of > > operator events that need to be transmitted between the coordinator and the > > writer operators? This information could help provide estimations on the > > additional workload to the JM, preventing the JM from being a single > > bottleneck to the throughput of Paimon sinks. > > > > Best, > > Yunfeng > > > > > 2025年1月23日 17:44，Yong Fang <zjur...@gmail.com> 写道： > > > > > > Hi devs, > > > > > > I would like to start a discussion about PIP-30: Improvement For Paimon > > > Committer In Flink [1]. > > > > > > Currently Flink writes data to Paimon based on Two-Phase Commit which will > > > generate a global committer node and connect all tasks in one region. If > > > any task fails, it will lead to a global failover in Flink job. > > > > > > To solve this issue, we would like to introduce a Paimon Writer > > > Coordinator > > > to perform table commit operation, enabling Flink paimon jobs to support > > > region failover and improving stability. > > > > > > Looking forward to hearing from you, thanks! > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/PAIMON/PIP-30%3A+Improvement+For+Paimon+Committer+In+Flink > > > > > > > > > Best, > > > Fang Yong > >
