Hi Gordon! Thank you for preparing the detailed FLIP, I think this is a huge improvement that enables the exactly-once Kafka sink in many environments / use-cases where this was previously unfeasible due to the limitations highlighted in the FLIP.
Big +1 Cheers, Gyula On Fri, Aug 18, 2023 at 7:54 PM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Hi Flink devs, > > I’d like to officially start a discussion for FLIP-319: Integrating with > Kafka’s proper support for 2PC participation (KIP-939) [1]. > > This is the “sister” joint FLIP for KIP-939 [2] [3]. It has been a > long-standing issue that Flink’s Kafka connector doesn’t work fully > correctly under exactly-once mode due to lack of distributed transaction > support in the Kafka transaction protocol. This has led to subpar hacks in > the connector such as Java reflections to workaround the protocol's > limitations (which causes a bunch of problems on its own, e.g. long > recovery times for the connector), while still having corner case scenarios > that can lead to data loss. > > This joint effort with the Kafka community attempts to address this so that > the Flink Kafka connector can finally work against public Kafka APIs, which > should result in a much more robust integration between the two systems, > and for Flink developers, easier maintainability of the code. > > Obviously, actually implementing this FLIP relies on the joint KIP being > implemented and released first. Nevertheless, I'd like to start the > discussion for the design as early as possible so we can benefit from the > new Kafka changes as soon as it is available. > > Looking forward to feedback and comments on the proposal! > > Thanks, > Gordon > > [1] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255071710 > [2] > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC > [3] https://lists.apache.org/thread/wbs9sqs3z1tdm7ptw5j4o9osmx9s41nf >
