We have a 3 broker Kafka 0.10.0.1 cluster. There we have 3 topics with 10 partitions each. We have an application which spawns threads as consumers. We spawn 5 consumers for each topic. I am observing that consider group randomly keeps rebalancing. Then many times we see logs saying "Revoking partitions for". This happens almost every 10 minutes. Consumption during this time completely stops.
I have applied this configuration max.poll.records 20 heartbeat.interval.ms 10000 Session.timeout.ms 6000 Still this did not help. Strange thing is I observed consumer writing logs saying "auto commit failed because poll() loop spent too much time processing records" even when there was no data in partition to process. We have polling interval of 500 ms, specified as argument in poll(). Initially I had set same consumer group for all three topics' consumers. Then I specified different CGs for different topics' consumers. Even this is not helping. I am trying to search over the web, checked my code, tried many combinations of configuration but still no luck. Please help me. Thanks & Regards, Shantanu Deshmukh
