I'm running kafka-0.7 on a hbase cluster where the zookeepers are from Cloudera's CHD3 Update 2 = zookeeper-3.3.3
thank you for you time and support :) I'll try and add a shutdown hook and close the consumer more "nicely". /Peter Den 20. mar. 2012 16.49 skrev Neha Narkhede <neha.narkh...@gmail.com>: > Peter, > > >> If this exception is thrown, will the consumer the intelligently wait > for the rebalancing to complete? and then resume consumption? > > If this exception is thrown, it means that the consumer has failed the > current rebalancing attempt and will try only when one of the > following happens - > > 1. New partitions are added to the topic it is consuming > 2. Existing partitions become unavailable > 3. New consumer instances are brought up for the consumer group it belongs > to > 4. Existing consumer instances die for the consumer group it belongs to > > Until that, the consumer is not fully functional. So, this particular > exception should be monitored and the consumer instance should be > restarted. > > Having said that, it is pretty rare for the consumer to run out of > rebalancing attempts. One of the common causes is using zookeeper > 3.3.3 which causes older ephemeral nodes to be retained. > Which version of Kafka are you using ? > Would you mind attaching the entire log for the consumer. It will help > us debug the cause of this exception and see if it is an actual bug. > > Thanks, > Neha > > > On Tue, Mar 20, 2012 at 2:42 AM, Peter Thygesen <pt.activ...@gmail.com> > wrote: > > When I shutdown my consumer with crtl-c and tries to restart it quickly > > afterwards, I usually get ConsumerRebalanceFailedException (see below). > The > > application then seems to hang.. or at least I'm sure if it is running > any > > more.. If this exception is thrown, will the consumer the intelligently > > wait for the rebalancing to complete? and then resume consumption? > > > > I found a page > https://cwiki.apache.org/KAFKA/consumer-co-ordinator.htmlthat > > describes something about Consumer Co-ordinator.. according to this > > the consumer > > group remains in this state until the next rebalancing attempt is > > triggered. But when is it triggered? > > > > Could a shutdown hock with a consumer.commitOffsets help? > > Does the consumer.shutdown implicit commitOffsets? > > > > > > Exception in thread "main" kafka.common.ConsumerRebalanceFailedException: > > contentItem-consumer-group-1_cphhdfs01node09-1332175323213-e6a3010f can't > > rebalance after 4 retries > > at > > > kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:467) > > at > > > kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:204) > > at > > > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:75) > > at > > > kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:89) > > at > > > com.infopaq.research.repository.uima.ContentItemClient.consume(ContentItemClient.java:75) > > at > > > com.infopaq.research.repository.uima.ContentItemClient.main(ContentItemClient.java:111) > > > > > > Brgds, > > Peter Thygesen > > > > BTW: Great work, very interesting project. >
