Hello everyone, We have a 3 broker Kafka 0.10.1.0 cluster in production environment. Lately we are seeing a lot of "auto commit failed because poll() spend too much time processing" warning messages. Also, due to such events there is constant fear of duplicate messages and the same does happen. To tackle this, I have tried a number of configuration combinations. I have set max.poll.records to as low as 5. Increased max.poll.interval.ms to 10 minutes. increased session.timeout.ms to 5 minutes etc. However, this has not helped. So, this has got me wondering shall we go after manual commit of offsets?
Our consumer process has one topic which has produce rate of about 15k events per second. It takes about 25ms to process one event of that topic and we have 5 consumers for that topic. If I do manual commit of every single event that we process what cost will it incur on host machine or network etc? Is there a better optimized method of manual commit? Or better yet, how to avoid "auto commit failed" error? *Thanks & Regards,* *Shantanu Deshmukh*
