Hi, In Flink, there is a TwoPhaseCommit SinkFunction that can be used for such cases: [1]. The PR for a Kafka 0.11 exactly once producer builds on that: [2]
Best, Aljoscha [1] https://github.com/apache/flink/blob/62e99918a45b7215c099fbcf160d45aa02d4559e/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/TwoPhaseCommitSinkFunction.java#L55 <https://github.com/apache/flink/blob/62e99918a45b7215c099fbcf160d45aa02d4559e/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/TwoPhaseCommitSinkFunction.java#L55> [2] https://github.com/apache/flink/pull/4239 > On 3. Aug 2017, at 04:03, Raghu Angadi <[email protected]> wrote: > > Kafka 0.11 added support for transactions[1], which allows end-to-end > exactly-once semantics. Beam's KafkaIO users can benefit from these while > using runners that support exactly-once processing. > > I have an implementation of EOS support for Kafka sink : > https://github.com/apache/beam/pull/3612 > It has two shuffles and builds on Beam state-API and checkpoint barrier > between stages (as in Dataflow). Pull request has a longer description. > > - What other runners in addition to Dataflow would be compatible with such > a strategy? > - I think it does not quite work for Flink (as it has a global checkpoint, > not between the stages). How would one go about implementing such a sink. > > Any comments on the pull request are also welcome. > > Thanks, > Raghu. > > [1] > https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/
