Re: Re: Re: Re: Re: Kafka lost data issue

If producer doesn't get a response, retries but both produce-requests succeeded, you will get duplicates. Kafka does not have a Idempotent Producer. On Fri, Nov 13, 2015 at 4:35 AM, Prabhjot Bharaj wrote: > Hi Gwen, > > If producer cant get a response but the message got

Re: Re: Re: Re: Re: Kafka lost data issue

Hi Gwen, If producer cant get a response but the message got committed, Because of retries, can this producer be committing messages more than once?? I'm trying to see this problem from the point of view of Idempotent Producer Thanks, Prabhjot On Nov 13, 2015 1:26 PM, "Hawin Jiang"

Re: Re: Re: Re: Re: Kafka lost data issue

Thanks Gwen for your excellent slides I will test it again based on your suggestions. Best regards Hawin On Thu, Nov 12, 2015 at 6:35 PM, Gwen Shapira wrote: > Hi, > > First, here's a handy slide-deck on avoiding data loss in Kafka: > >

Re: Re: Re: Re: Re: Kafka lost data issue

Hi, First, here's a handy slide-deck on avoiding data loss in Kafka: http://www.slideshare.net/gwenshap/kafka-reliability-when-it-absolutely-positively-has-to-be-there Note configuration parameters like the number of retries. Also, it looks like you are sending data to Kafka asynchronously, but

Re: Re: Re: Re: Re: Kafka lost data issue

Hi Pradeep Here is my configuration # Producer Basics # # list of brokers used for bootstrapping knowledge about the rest of the cluster # format: host1:port1,host2:port2 ... metadata.broker.list=localhost:9092 # name of the partitioner