There are some details to get this right, the lookup table has to survive failures. But yes this is exactly what we would like to add:
https://cwiki.apache.org/confluence/display/KAFKA/Idempotent+Producer -Jay On Tue, Feb 17, 2015 at 12:44 AM, Arunkumar Srambikkal (asrambik) < asram...@cisco.com> wrote: > Hi, > > I guess message production duplicate scenario in Kafka is, when a producer > commits the data but does not get an ack (broker or network fails AFTER > commit) and retries. > > I got thinking that the retry can be caught by the broker which could then > identify the previous message with a unique message ID before committing > them again but it requires a small lookup table for the previous commit ID > for each producer. > > Can I please know what is wrong about this picture, since Kafka does not > do this? > > Thanks > Arun >
