Re: I found kafka lsot message
Watch this talk. Kafka will not lose messages it configured correctly. http://www.confluent.io/kafka-summit-2016-ops-when-it-absolutely-positively-has-to-be-there <http://www.confluent.io/kafka-summit-2016-ops-when-it-absolutely-positively-has-to-be-there> -hans > On Oct 13, 2016, at 2:48 AM, wrote: > > Other question: if account of consumer over broker number, whether loss > data?if some broker rebooted, whether loss data? > Thanks. > > > > > - 原始邮件 - > 发件人:Guozhang Wang > 收件人:yangy...@sina.com > 抄送人:users > 主题:Re: Re: I found kafka lsot message > 日期:2016年10月12日 07点05分 > > One common issue of lost messages is that consumer auto-committing (related > config is "auto.commit.enabled", "commit.interval.ms"): from the Kafka > consumer point of view, once the messages are returned from the "poll" call > they are considered "consumed", and if committing offsets is called it will > set the offset to the last message's offset. So if you have some exceptions / > errors in the middle of processing some of your consumed records, upon > restarting they will not be re-fetched again, hence possibly causing "data > loss" on your end. > > Guozhang > On Mon, Oct 10, 2016 at 3:33 AM, wrote: > Hi Guozhang, > At first, thank you answer my question, and give me some suggest.But, I'm > sure I readed some introduction about kafka. > In my producer, My Code is( c code):res = rd_kafka_conf_set(kafka_conf, > "request.required.acks", "-1", NULL, 0);res = rd_kafka_topic_conf_set( > topic_conf, "produce.offset.report", "true", errstr, sizeof(errstr) ); > In my consumer, My Code is(kafka-python): >self.__consumer = KafkaConsumer( > bootstrap_servers=self.__kafka_config["hosts"], > group_id=self.__kafka_config["group"], > auto_offset_reset="earliest", > ) > self.__consumer.subscribe([self.__kafka_config["data_topic"]]) >for message in self.__consumer: > > I think these codes is common, What's your suggest about these codes? > In the end, I must explain: Lost message is not often, some time, couple days > can find one or two lost messages.But some day, maybe can find over 20 > messages were losted.Our message over 1,200,000 in one day. > So, do your have any suggest?Bye the way, can you speak Chinese? Thank you > very much & Best wishesJerry > > > > > - 原始邮件 - > 发件人:Guozhang Wang > 收件人:"users@kafka.apache.org" , yangy...@sina.com > 主题:Re: I found kafka lsot message > 日期:2016年10月10日 01点25分 > > Jerry, > Message lost scenarios usually are due to producer and consumer > mis-configured. Have you read about the client configs web docs? > http://kafka.apache.org/documentation#producerconfigs > > http://kafka.apache.org/documentation#newconsumerconfigs > > > If not I'd suggest you reading those first and see if you can tune some of > these configs to have better delivery guarantees. > Guozhang > > On Fri, Oct 7, 2016 at 9:48 PM, wrote: > Hello every body,I build a kafka cluster(include 5 domains) use > kafka_2.11-0.10.0.0 and kafka-python-1.3.1.I create a topic by 100 partitions > and 2 replicate, then use 20 consumers to receive message. > > But, I found sometime the kafka lost message! I found some partition's > offsite lost at consumer.After, I make a global index for every message from > producer for confirm this problem, and I also found the global index had been > break! > > Why the kafka lost message? What can I do to fix the problem? > > Thanks!Jerry > > > > > > > > > > -- > -- Guozhang > > > > > > -- > -- Guozhang > > >
回复:Re: Re: I found kafka lsot message
Other question: if account of consumer over broker number, whether loss data? if some broker rebooted, whether loss data? Thanks. - 原始邮件 - 发件人:Guozhang Wang 收件人:yangy...@sina.com 抄送人:users 主题:Re: Re: I found kafka lsot message 日期:2016年10月12日 07点05分 One common issue of lost messages is that consumer auto-committing (related config is "auto.commit.enabled", "commit.interval.ms"): from the Kafka consumer point of view, once the messages are returned from the "poll" call they are considered "consumed", and if committing offsets is called it will set the offset to the last message's offset. So if you have some exceptions / errors in the middle of processing some of your consumed records, upon restarting they will not be re-fetched again, hence possibly causing "data loss" on your end. Guozhang On Mon, Oct 10, 2016 at 3:33 AM, wrote: Hi Guozhang, At first, thank you answer my question, and give me some suggest.But, I'm sure I readed some introduction about kafka. In my producer, My Code is( c code):res = rd_kafka_conf_set(kafka_conf, "request.required.acks", "-1", NULL, 0);res = rd_kafka_topic_conf_set( topic_conf, "produce.offset.report", "true", errstr, sizeof(errstr) ); In my consumer, My Code is(kafka-python): self.__consumer = KafkaConsumer( bootstrap_servers=self.__kafka_config["hosts"], group_id=self.__kafka_config["group"], auto_offset_reset="earliest", ) self.__consumer.subscribe([self.__kafka_config["data_topic"]]) for message in self.__consumer: I think these codes is common, What's your suggest about these codes? In the end, I must explain: Lost message is not often, some time, couple days can find one or two lost messages.But some day, maybe can find over 20 messages were losted.Our message over 1,200,000 in one day. So, do your have any suggest?Bye the way, can you speak Chinese? Thank you very much & Best wishesJerry - 原始邮件 - 发件人:Guozhang Wang 收件人:"users@kafka.apache.org" , yangy...@sina.com 主题:Re: I found kafka lsot message 日期:2016年10月10日 01点25分 Jerry, Message lost scenarios usually are due to producer and consumer mis-configured. Have you read about the client configs web docs? http://kafka.apache.org/documentation#producerconfigs http://kafka.apache.org/documentation#newconsumerconfigs If not I'd suggest you reading those first and see if you can tune some of these configs to have better delivery guarantees. Guozhang On Fri, Oct 7, 2016 at 9:48 PM, wrote: Hello every body,I build a kafka cluster(include 5 domains) use kafka_2.11-0.10.0.0 and kafka-python-1.3.1.I create a topic by 100 partitions and 2 replicate, then use 20 consumers to receive message. But, I found sometime the kafka lost message! I found some partition's offsite lost at consumer.After, I make a global index for every message from producer for confirm this problem, and I also found the global index had been break! Why the kafka lost message? What can I do to fix the problem? Thanks!Jerry -- -- Guozhang -- -- Guozhang
Re: Re: I found kafka lsot message
One common issue of lost messages is that consumer auto-committing (related config is "auto.commit.enabled", "commit.interval.ms"): from the Kafka consumer point of view, once the messages are returned from the "poll" call they are considered "consumed", and if committing offsets is called it will set the offset to the last message's offset. So if you have some exceptions / errors in the middle of processing some of your consumed records, upon restarting they will not be re-fetched again, hence possibly causing "data loss" on your end. Guozhang On Mon, Oct 10, 2016 at 3:33 AM, wrote: > Hi Guozhang, > At first, thank you answer my question, and give me some suggest. > But, I'm sure I readed some introduction about kafka. > > In my producer, My Code is( c code): > res = rd_kafka_conf_set(kafka_conf, "request.required.acks", "-1", NULL, > 0); > res = rd_kafka_topic_conf_set( topic_conf, "produce.offset.report", > "true", errstr, sizeof(errstr) ); > > In my consumer, My Code is(kafka-python): > > self.__consumer = KafkaConsumer( bootstrap_servers=self.__ > kafka_config["hosts"], > group_id=self.__kafka_config[" > group"], > auto_offset_reset="earliest", > ) > self.__consumer.subscribe([self.__kafka_config["data_ > topic"]]) > > for message in self.__consumer: > > > I think these codes is common, What's your suggest about these codes? > > In the end, I must explain: Lost message is not often, some time, couple > days can find one or two lost messages. > But some day, maybe can find over 20 messages were losted. > Our message over 1,200,000 in one day. > > So, do your have any suggest? > Bye the way, can you speak Chinese? > Thank you very much & Best wishes > Jerry > > > > > > - 原始邮件 - > 发件人:Guozhang Wang > 收件人:"users@kafka.apache.org" , yangy...@sina.com > 主题:Re: I found kafka lsot message > 日期:2016年10月10日 01点25分 > > Jerry, > > Message lost scenarios usually are due to producer and consumer > mis-configured. Have you read about the client configs web docs? > > http://kafka.apache.org/documentation#producerconfigs > > http://kafka.apache.org/documentation#newconsumerconfigs > > > If not I'd suggest you reading those first and see if you can tune some of > these configs to have better delivery guarantees. > > Guozhang > > > On Fri, Oct 7, 2016 at 9:48 PM, wrote: > > Hello every body,I build a kafka cluster(include 5 domains) use > kafka_2.11-0.10.0.0 and kafka-python-1.3.1.I create a topic by 100 > partitions and 2 replicate, then use 20 consumers to receive message. > But, I found sometime the kafka lost message! I found some partition's > offsite lost at consumer.After, I make a global index for every message > from producer for confirm this problem, and I also found the global index > had been break! > Why the kafka lost message? What can I do to fix the problem? > Thanks!Jerry > > > > > > > -- > -- Guozhang > -- -- Guozhang
回复:Re: I found kafka lsot message
Hi Guozhang, At first, thank you answer my question, and give me some suggest.But, I'm sure I readed some introduction about kafka. In my producer, My Code is( c code):res = rd_kafka_conf_set(kafka_conf, "request.required.acks", "-1", NULL, 0);res = rd_kafka_topic_conf_set( topic_conf, "produce.offset.report", "true", errstr, sizeof(errstr) ); In my consumer, My Code is(kafka-python): self.__consumer = KafkaConsumer( bootstrap_servers=self.__kafka_config["hosts"], group_id=self.__kafka_config["group"], auto_offset_reset="earliest", ) self.__consumer.subscribe([self.__kafka_config["data_topic"]]) for message in self.__consumer: I think these codes is common, What's your suggest about these codes? In the end, I must explain: Lost message is not often, some time, couple days can find one or two lost messages.But some day, maybe can find over 20 messages were losted.Our message over 1,200,000 in one day. So, do your have any suggest?Bye the way, can you speak Chinese? Thank you very much & Best wishesJerry ---- - 原始邮件 - 发件人:Guozhang Wang 收件人:"users@kafka.apache.org" , yangy...@sina.com 主题:Re: I found kafka lsot message 日期:2016年10月10日 01点25分 Jerry, Message lost scenarios usually are due to producer and consumer mis-configured. Have you read about the client configs web docs? http://kafka.apache.org/documentation#producerconfigs http://kafka.apache.org/documentation#newconsumerconfigs If not I'd suggest you reading those first and see if you can tune some of these configs to have better delivery guarantees. Guozhang On Fri, Oct 7, 2016 at 9:48 PM, wrote: Hello every body,I build a kafka cluster(include 5 domains) use kafka_2.11-0.10.0.0 and kafka-python-1.3.1.I create a topic by 100 partitions and 2 replicate, then use 20 consumers to receive message. But, I found sometime the kafka lost message! I found some partition's offsite lost at consumer.After, I make a global index for every message from producer for confirm this problem, and I also found the global index had been break! Why the kafka lost message? What can I do to fix the problem? Thanks!Jerry -- -- Guozhang
Re: I found kafka lsot message
Jerry, Message lost scenarios usually are due to producer and consumer mis-configured. Have you read about the client configs web docs? http://kafka.apache.org/documentation#producerconfigs http://kafka.apache.org/documentation#newconsumerconfigs If not I'd suggest you reading those first and see if you can tune some of these configs to have better delivery guarantees. Guozhang On Fri, Oct 7, 2016 at 9:48 PM, wrote: > Hello every body,I build a kafka cluster(include 5 domains) use > kafka_2.11-0.10.0.0 and kafka-python-1.3.1.I create a topic by 100 > partitions and 2 replicate, then use 20 consumers to receive message. > But, I found sometime the kafka lost message! I found some partition's > offsite lost at consumer.After, I make a global index for every message > from producer for confirm this problem, and I also found the global index > had been break! > Why the kafka lost message? What can I do to fix the problem? > Thanks!Jerry > > > > -- -- Guozhang