That's awesome! I will look into it more this weekend.
On Tue, Aug 8, 2017 at 8:22 AM, Aljoscha Krettek <aljos...@apache.org>
> In Flink, there is a TwoPhaseCommit SinkFunction that can be used for such
> cases: . The PR for a Kafka 0.11 exactly once producer builds on that:
>  https://github.com/apache/flink/blob/62e99918a45b7215c099fbcf160d45
> flink/streaming/api/functions/sink/TwoPhaseCommitSinkFunction.java#L55 <
>  https://github.com/apache/flink/pull/4239
> > On 3. Aug 2017, at 04:03, Raghu Angadi <rang...@google.com.INVALID>
> > Kafka 0.11 added support for transactions, which allows end-to-end
> > exactly-once semantics. Beam's KafkaIO users can benefit from these while
> > using runners that support exactly-once processing.
> > I have an implementation of EOS support for Kafka sink :
> > https://github.com/apache/beam/pull/3612
> > It has two shuffles and builds on Beam state-API and checkpoint barrier
> > between stages (as in Dataflow). Pull request has a longer description.
> > - What other runners in addition to Dataflow would be compatible with
> > a strategy?
> > - I think it does not quite work for Flink (as it has a global
> > not between the stages). How would one go about implementing such a sink.
> > Any comments on the pull request are also welcome.
> > Thanks,
> > Raghu.
> > 
> > https://www.confluent.io/blog/exactly-once-semantics-are-