Exactly once Kafka Sink Connectors typically store the offset externally in the same atomic write as they store the messages. That way after a crash, they can check the external store (HSFS, JSDC, etc) retrieve the last committed offset and seek the the next message and continue processing with no duplicates and exactly once semantics.
-hans > On Dec 21, 2016, at 4:39 PM, kant kodali <[email protected]> wrote: > > How does Kafka emulate exactly once processing currently? Does it require > the producer to send at least once and consumer to de dupe? > > I did do my research but I feel like I am going all over the place so a > simple short answer would be great! > > Thanks!
