Sorry, it seems that we misconfigured our Zookeeper cluster, and it was acting as individual nodes rather than one cluster. That was probably it. :-p
On Wed, Jan 11, 2012 at 1:30 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > Evan, > > What you are describing seems to be this bug in zookeeper 3.3.3 - > https://issues.apache.org/jira/browse/ZOOKEEPER-1208 > > However, to confirm if thats the case, do you mind uploading the > entire log for the consumer ? > > Thanks, > Neha > > On Wed, Jan 11, 2012 at 11:33 AM, Evan Chan <e...@ooyala.com> wrote: > > Oh, here are the consumer logs when it can't claim a partition: > > > > > > 2012-01-11 02:23:04 ZookeeperConsumerConnector [WARN] No broker partions > > consumed by consumer thread > > whatever_ip-10-116-81-39.ec2.internal-1326248583706-0 > > for t > > opic player_logs > > 2012-01-11 02:23:04 ZookeeperConsumerConnector [INFO] Consumer > > whatever_ip-10-116-81-39.ec2.internal-1326248583706 selected partitions : > > > > > > On Wed, Jan 11, 2012 at 11:32 AM, Evan Chan <e...@ooyala.com> wrote: > > > >> Hi, > >> > >> We are using Kafka 0.6 and testing it on EC2. We have an issue where > some > >> processes running the ZK/High level consumer (Scala consumer) get killed > >> before they have a chance to call ConsumerConnector.shutdown(). > >> It seems like they leave nodes hanging around in ZK. > >> If we restart the process, the consumer in the new process will error > out > >> because it cannot claim any partitions. > >> > >> The only way I know of getting around this is to use a ZK client and > >> manually delete nodes. > >> > >> Is there any way for the high level consumer nodes in ZK to be made > >> ephemeral so that if a process gets killed, the state won't last forever > >> and cause subsequent nodes to not be able to claim partitions? > >> Any chance this has been fixed in 0.7? > >> > >> thanks, > >> Evan > >> > >> -- > >> -- > >> *Evan Chan* > >> Senior Software Engineer | > >> e...@ooyala.com | (650) 996-4600 > >> www.ooyala.com | blog <http://www.ooyala.com/blog> | @ooyala< > http://www.twitter.com/ooyala> > >> > > > > > > > > -- > > -- > > *Evan Chan* > > Senior Software Engineer | > > e...@ooyala.com | (650) 996-4600 > > www.ooyala.com | blog <http://www.ooyala.com/blog> | > > @ooyala<http://www.twitter.com/ooyala> > -- -- *Evan Chan* Senior Software Engineer | e...@ooyala.com | (650) 996-4600 www.ooyala.com | blog <http://www.ooyala.com/blog> | @ooyala<http://www.twitter.com/ooyala>
