I am trying to replace ActiveMQ with Kafka in our environment however I have encountered a strange problem that basically prevents from using Kafka in production. The problem is that sometimes the offsets are not committed.
I am using Kafka 0.8.2.1, offset storage = kafka, high level consumer, auto-commit = off. Every N messages I issue commitOffsets(). Now here is the problem - if N is below a certain number (180 000 for me) it works and the offset is moving. If N is 180 000 or more the offset is not updated after commitOffsets I am looking at offsets using kafka-run-class.sh kafka.tools.ConsumerOffsetChecker Any help?
