Hi, We have a Kafka cluster (version 1.1.1) where one node unexpectedly failed. After that consumers from a couple of consumers are stuck in the poll() API call. Looking at the thread dump, it looks like the consumer is stuck in org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady() call. The heartbeat thread is also blocked waiting for the ConsumerCoordinator. Any idea what is the cause and how to resolve this issue? Thanks.
"BusinessEventRecordsDispatcherThread" #43 prio=5 os_prio=0 tid=0x00007f71764fb800 nid=0x241e sleeping[0x00007f70d24e9000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at org.apache.kafka.common.utils.SystemTime.sleep(SystemTime.java:45) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:235) - locked <0x00000004fc30e628> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:205) - locked <0x00000004fc30e628> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:351) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:316) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1149) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115) ... at java.lang.Thread.run(Thread.java:748) "kafka-coordinator-heartbeat-thread | prod-mkt-datahub-loader" #45 daemon prio=5 os_prio=0 tid=0x00007f711a286800 nid=0x2422 in Object.wait() [0x00007f7120125000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000004fc30e628> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) at java.lang.Object.wait(Object.java:502) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:937) - locked <0x00000004fc30e628> (a org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)