So here, when I received a message I run some business logic on it and try to send some email. Now sometimes we have a promotional campaign running millions of emails need to be delivered. For such numerous events is manual commit good? Will it generate too much network activity if I commit a single message every time?
On Thu, May 24, 2018 at 6:20 PM M. Manna <manme...@gmail.com> wrote: > Manual commit is important where event consumption eventually leads to some > post-processing/database update/state change for your application. Without > doing all those, you cannot truly say that you have "Received" the message. > "Receiving" is interpreted differently and it's up to your target > application. > > Jason Gustafson has written some descriptive post regarding transactions. I > don't know if you can extrapolate your changes based on those: > > 1) https://www.confluent.io/blog/transactions-apache-kafka/ > 2) > > https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/ > > ALso, please don't forget to read Javadoc on KafkaConsumer.java > > Regards, > > On 24 May 2018 at 13:29, Shantanu Deshmukh <shantanu...@gmail.com> wrote: > > > Hello everyone, > > > > We have a 3 broker Kafka 0.10.1.0 cluster in production environment. > Lately > > we are seeing a lot of "auto commit failed because poll() spend too much > > time processing" warning messages. Also, due to such events there is > > constant fear of duplicate messages and the same does happen. To tackle > > this, I have tried a number of configuration combinations. I have set > > max.poll.records to as low as 5. Increased max.poll.interval.ms to 10 > > minutes. increased session.timeout.ms to 5 minutes etc. However, this > has > > not helped. So, this has got me wondering shall we go after manual commit > > of offsets? > > > > Our consumer process has one topic which has produce rate of about 15k > > events per second. It takes about 25ms to process one event of that topic > > and we have 5 consumers for that topic. If I do manual commit of every > > single event that we process what cost will it incur on host machine or > > network etc? Is there a better optimized method of manual commit? Or > better > > yet, how to avoid "auto commit failed" error? > > > > *Thanks & Regards,* > > *Shantanu Deshmukh* > > >
