If the consumer is cleanly shutdown (Control-C is kill, not kill -9, right?), shoudn't we remove the ephemeral ZNodes in our shutdown handler? I thought this was already the case, at least for the broker.
On 03/20/2012 12:21 PM, Jun Rao wrote: > Peter, > > When you kill your consumer, it takes sometime (by default 6 secs) for ZK > server to release the ephemeral nodes hold by the consumer. If you restart > your consumer quickly, the consumer may not be able to acquire the > necessary zk nodes and therefore can fail to rebalance. One way is to > implement a shutdown hook in your consumer application and make sure > consumer connector is shut down when the consumer is killed. > > Thanks, > > Jun > > On Tue, Mar 20, 2012 at 2:42 AM, Peter Thygesen <pt.activ...@gmail.com>wrote: > >> When I shutdown my consumer with crtl-c and tries to restart it quickly >> afterwards, I usually get ConsumerRebalanceFailedException (see below). The >> application then seems to hang.. or at least I'm sure if it is running any >> more.. If this exception is thrown, will the consumer the intelligently >> wait for the rebalancing to complete? and then resume consumption? >> >> I found a page >> https://cwiki.apache.org/KAFKA/consumer-co-ordinator.htmlthat >> describes something about Consumer Co-ordinator.. according to this >> the consumer >> group remains in this state until the next rebalancing attempt is >> triggered. But when is it triggered? >> >> Could a shutdown hock with a consumer.commitOffsets help? >> Does the consumer.shutdown implicit commitOffsets? >> >> >> Exception in thread "main" kafka.common.ConsumerRebalanceFailedException: >> contentItem-consumer-group-1_cphhdfs01node09-1332175323213-e6a3010f can't >> rebalance after 4 retries >> at >> >> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:467) >> at >> >> kafka.consumer.ZookeeperConsumerConnector.consume(ZookeeperConsumerConnector.scala:204) >> at >> >> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:75) >> at >> >> kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:89) >> at >> >> com.infopaq.research.repository.uima.ContentItemClient.consume(ContentItemClient.java:75) >> at >> >> com.infopaq.research.repository.uima.ContentItemClient.main(ContentItemClient.java:111) >> >> >> Brgds, >> Peter Thygesen >> >> BTW: Great work, very interesting project. >> >
