Re: Re: Re: Re: Re: Kafka lost data issue

If producer doesn't get a response, retries but both produce-requests succeeded, you will get duplicates. Kafka does not have a Idempotent Producer. On Fri, Nov 13, 2015 at 4:35 AM, Prabhjot Bharaj wrote: > Hi Gwen, > > If producer cant get a response but the message got

Re: Re: Re: Re: Re: Kafka lost data issue

Hi Gwen, If producer cant get a response but the message got committed, Because of retries, can this producer be committing messages more than once?? I'm trying to see this problem from the point of view of Idempotent Producer Thanks, Prabhjot On Nov 13, 2015 1:26 PM, "Hawin Jiang"

Re: Re: Kafka lost data issue

Hi Jinxing I don't think we can resolve this issue by increasing producers. if I increased more producers, it should lost more messages. I just test two producers. Thread Producer 1 has 83064 messages in producer side and 82273 messages in consumer side Thread Producer 2 has 89844 messages in

Re:Re: Re: Kafka lost data issue

there is a flush api of the producer, you can call this to prevent messages lost; maybe it can help; At 2015-11-12 16:43:54, "Hawin Jiang" wrote: >Hi Jinxing > >I don't think we can resolve this issue by increasing producers. if I >increased more producers, it

Re:Re: Re: Re: Kafka lost data issue

in kafka_0.8.3.0: kafkaProducer = new KafkaProducer<>(properties, new ByteArraySerializer(), new ByteArraySerializer()); kafkaProducer.flush(); you can call the flush after sending every few messages; At 2015-11-12 17:36:24, "Hawin Jiang" wrote: >Hi Prabhjot > >The

Re: Re: Kafka lost data issue

Hi, Just to confirm that the number of messages produced are what you are seeing, What does GetOffsetShell report for this topic ? Regards, Prabhjot On Thu, Nov 12, 2015 at 2:13 PM, Hawin Jiang wrote: > Hi Jinxing > > I don't think we can resolve this issue by

Re: Re: Re: Kafka lost data issue

Hi Prabhjot The messages are "Thread1_kafka_1" and "Thread2_kafka_1". Something like that. For GetOffsetShell report below: [kafka@dn-01 bin]$ ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list dn-01:9092 --time -1 --topic kafka-test kafka-test:0:12529261 @Jinxing Can you share

Re: Kafka lost data issue

Hi Jinxing I am using kafka_2.10-0.9.0.0-SNAPSHOT. I have downloaded source code and installed it last week. I saw 97446 messages have been sent to kafka successfully. So far, I have not found any failed messages. Best regards Hawin On Thu, Nov 12, 2015 at 12:07 AM, jinxing

Re: Re: Re: Re: Re: Kafka lost data issue

Thanks Gwen for your excellent slides I will test it again based on your suggestions. Best regards Hawin On Thu, Nov 12, 2015 at 6:35 PM, Gwen Shapira wrote: > Hi, > > First, here's a handy slide-deck on avoiding data loss in Kafka: > >

Re: Re: Re: Re: Re: Kafka lost data issue

Hi, First, here's a handy slide-deck on avoiding data loss in Kafka: http://www.slideshare.net/gwenshap/kafka-reliability-when-it-absolutely-positively-has-to-be-there Note configuration parameters like the number of retries. Also, it looks like you are sending data to Kafka asynchronously, but

Re:Re: Re: Re: Re: Kafka lost data issue

i have 3 brokers; the ack configuration is -1(all), meaning a message is sent successfully only after getting every broker's ack; is this a bug? At 2015-11-12 21:08:49, "Pradeep Gollakota" wrote: >What is your producer configuration? Specifically, how many acks are

Re: Re: Re: Re: Re: Kafka lost data issue

Hi Pradeep Here is my configuration # Producer Basics # # list of brokers used for bootstrapping knowledge about the rest of the cluster # format: host1:port1,host2:port2 ... metadata.broker.list=localhost:9092 # name of the partitioner