On Mar 20, 2012, at 8:49 AM, Neha Narkhede wrote: > Peter, > >>> If this exception is thrown, will the consumer the intelligently wait for >>> the rebalancing to complete? and then resume consumption? > > If this exception is thrown, it means that the consumer has failed the > current rebalancing attempt and will try only when one of the > following happens - > > 1. New partitions are added to the topic it is consuming > 2. Existing partitions become unavailable > 3. New consumer instances are brought up for the consumer group it belongs to > 4. Existing consumer instances die for the consumer group it belongs to > > Until that, the consumer is not fully functional. So, this particular > exception should be monitored and the consumer instance should be > restarted. > > Having said that, it is pretty rare for the consumer to run out of > rebalancing attempts. One of the common causes is using zookeeper > 3.3.3 which causes older ephemeral nodes to be retained. > Which version of Kafka are you using ? > Would you mind attaching the entire log for the consumer. It will help > us debug the cause of this exception and see if it is an actual bug. > > Thanks, > Neha > >
Neha, I see this exception 2012-07-17 12:58:12,238 ERROR [ZkClient-EventThread-17-11.zookeeper.,12.zookeeper.,13.zookeeper.,14.zookeeper.,16.zookeeper./kafka] zkclient.ZkEventThread Error handling event ZkEvent[Children of /consumers/live-event-sense-new8/ids changed sent to kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener@6d9dd520] java.lang.RuntimeException: live-event-sense-new8_sv4r25s49-1342554132312-c04abfef can't rebalance after 4 retires occurring very often. I use ZK 3.4.3. Im not handling/monitoring this exception . The consumer seems to continue just fine after this happens. I do not see any on the 4 conditions you mentioned happening. Am I missing something ? Thanks, Sam > On Tue, Mar 20, 2012 at 2:42 AM, Peter Thygesen <pt.activ...@gmail.com> wrote: >> When I shutdown my consumer with crtl-c and tries to restart it quickly >> afterwards, I usually get ConsumerRebalanceFailedException (see below). The >> application then seems to hang.. or at least I'm sure if it is running any >> more.. If this exception is thrown, will the consumer the intelligently >> wait for the rebalancing to complete? and then resume consumption? >> >> I found a page https://cwiki.apache.org/KAFKA/consumer-co-ordinator.htmlthat >> describes something about Consumer Co-ordinator.. according to this >> the consumer >> group remains in this state until the next rebalancing attempt is >> triggered. But when is it triggered? >> >> Could a shutdown hock with a consumer.commitOffsets help? >> Does the consumer.shutdown implicit commitOffsets? >> >> >> Exception in thread "main" kafka.common.ConsumerRebalanceFailedException: >> contentItem-consumer-group-1_cphhdfs01node09-1332175323213-e6a3010f can't >> rebalance after 4 retries >> at >> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:467) >> at >> kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:204) >> at >> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:75) >> at >> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:89) >> at >> com.infopaq.research.repository.uima.ContentItemClient.consume(ContentItemClient.java:75) >> at >> com.infopaq.research.repository.uima.ContentItemClient.main(ContentItemClient.java:111) >> >> >> Brgds, >> Peter Thygesen >> >> BTW: Great work, very interesting project. Sam William sa...@stumbleupon.com
