Peter, >> If this exception is thrown, will the consumer the intelligently wait for >> the rebalancing to complete? and then resume consumption?
If this exception is thrown, it means that the consumer has failed the current rebalancing attempt and will try only when one of the following happens - 1. New partitions are added to the topic it is consuming 2. Existing partitions become unavailable 3. New consumer instances are brought up for the consumer group it belongs to 4. Existing consumer instances die for the consumer group it belongs to Until that, the consumer is not fully functional. So, this particular exception should be monitored and the consumer instance should be restarted. Having said that, it is pretty rare for the consumer to run out of rebalancing attempts. One of the common causes is using zookeeper 3.3.3 which causes older ephemeral nodes to be retained. Which version of Kafka are you using ? Would you mind attaching the entire log for the consumer. It will help us debug the cause of this exception and see if it is an actual bug. Thanks, Neha On Tue, Mar 20, 2012 at 2:42 AM, Peter Thygesen <pt.activ...@gmail.com> wrote: > When I shutdown my consumer with crtl-c and tries to restart it quickly > afterwards, I usually get ConsumerRebalanceFailedException (see below). The > application then seems to hang.. or at least I'm sure if it is running any > more.. If this exception is thrown, will the consumer the intelligently > wait for the rebalancing to complete? and then resume consumption? > > I found a page https://cwiki.apache.org/KAFKA/consumer-co-ordinator.htmlthat > describes something about Consumer Co-ordinator.. according to this > the consumer > group remains in this state until the next rebalancing attempt is > triggered. But when is it triggered? > > Could a shutdown hock with a consumer.commitOffsets help? > Does the consumer.shutdown implicit commitOffsets? > > > Exception in thread "main" kafka.common.ConsumerRebalanceFailedException: > contentItem-consumer-group-1_cphhdfs01node09-1332175323213-e6a3010f can't > rebalance after 4 retries > at > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:467) > at > kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:204) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:75) > at > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:89) > at > com.infopaq.research.repository.uima.ContentItemClient.consume(ContentItemClient.java:75) > at > com.infopaq.research.repository.uima.ContentItemClient.main(ContentItemClient.java:111) > > > Brgds, > Peter Thygesen > > BTW: Great work, very interesting project.
