Re: ConsumerRebalanceFailedException with the kafka-console-consumer (bug?)

w.com/questions/36313470/consumerrebalancefailedexception-with-the-kafka-console-consumer > > Thanks, > Filipe > > On Wed, Mar 30, 2016 at 4:03 PM, Filipe Correia > wrote: > > > Hi there, > > > > I've just installed kafka 0.9.0.1, and I'm getting the following error &

Re: ConsumerRebalanceFailedException with the kafka-console-consumer (bug?)

I've also asked on stackoverflow, in case you prefer to answer there: http://stackoverflow.com/questions/36313470/consumerrebalancefailedexception-with-the-kafka-console-consumer Thanks, Filipe On Wed, Mar 30, 2016 at 4:03 PM, Filipe Correia wrote: > Hi there, > > I've j

ConsumerRebalanceFailedException with the kafka-console-consumer (bug?)

Hi there, I've just installed kafka 0.9.0.1, and I'm getting the following error when launching the kafka-console-consumer: $ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic myrandomtesttopic --from-beginning [2016-03-30 15:46:17,568] ERROR Unknown error when running consumer: (

Re: Kafka ConsumerRebalanceFailedException, kafka.consumer.SimpleConsumer: Reconnect due to socket error: null, etc.

One way to narrow down the issue is to attach a debugger to the Kafka JVM and add a breakpoint in SimpleConsumer to see the real exception stacktrace which is causing the reconnect. I've filed a JIRA with a patch to improve this logging to include the entire cause stacktrace while logging this

Kafka ConsumerRebalanceFailedException, kafka.consumer.SimpleConsumer: Reconnect due to socket error: null, etc.

Hi, I'm having trouble with a spark (1.2.1) streaming job which is using apache kafka (client: 0.8.1.1). The job is consuming messages, and the observed behaviour is that after processing some messages, the job stops processing, then restarts, then consumes the lag which build up during the pause

Re: Repeated failures due to ConsumerRebalanceFailedException

Apologies for not replying sooner. (I have digest subscription and I cannot reply to the email chain I started! So I had not received any direct replies either). @Mayuresh and @Jiangjie, there was nothing very indicative in the Kafka logs. But I'm writing to tell you that the issue was "resolved"

Re: Repeated failures due to ConsumerRebalanceFailedException

Do you see "zookeeper state changed (Expired)" in your logs? On Fri, Feb 27, 2015 at 10:12 AM, Jiangjie Qin wrote: > Can you paste the error log for each rebalance try? > You may search for keyword ³exception during rebalance². > > On 2/26/15, 7:41 PM, "Ashwin Jayaprakash" > wrote: > > >Just gi

Re: Repeated failures due to ConsumerRebalanceFailedException

Can you paste the error log for each rebalance try? You may search for keyword ³exception during rebalance². On 2/26/15, 7:41 PM, "Ashwin Jayaprakash" wrote: >Just give you some more debugging context, we noticed that the "consumers" >path becomes empty after all the JVMs have exited because of

Re: Repeated failures due to ConsumerRebalanceFailedException

Just give you some more debugging context, we noticed that the "consumers" path becomes empty after all the JVMs have exited because of this error. So, when we restart, there are no visible entries in ZK. On Thu, Feb 26, 2015 at 6:04 PM, Ashwin Jayaprakash < ashwin.jayaprak...@gmail.com> wrote: >

Re: Repeated failures due to ConsumerRebalanceFailedException

Some times an ephemeral ZK path does not go away after a consumer is closed. You can check the log for each rebalance to see if it complains some conflict data of ZK Path. If all the complaints are pointing to the same consumer, bounce that consumer. Otherwise you can try to remove the ZK path manu

Repeated failures due to ConsumerRebalanceFailedException

Hello, we have a set of JVMs that consume messages from Kafka topics. Each JVM creates 4 ConsumerConnectors that are used by 4 separate threads. These JVMs also create and use the CuratorFramework's Path children cache to watch and keep a sub-tree of the ZooKeeper in sync with other JVMs. This path

Re: How to recover from ConsumerRebalanceFailedException ?

Bhavesh, In 0.9 consumer would not talk to ZK and will be single threaded, which will be easier to provide monitoring mechanisms. Guozhang On Thu, Nov 20, 2014 at 8:15 PM, Jun Rao wrote: > Can you just monitor the consumer byte/message/fetch rate? > > Thanks, > > Jun > > On Thu, Nov 20, 2014 a

Re: How to recover from ConsumerRebalanceFailedException ?

Can you just monitor the consumer byte/message/fetch rate? Thanks, Jun On Thu, Nov 20, 2014 at 5:31 PM, Bhavesh Mistry wrote: > HI Jun, > > Do you want me to request Jira ticket for feature a notification for new > consumer API and old consumer feature that consumer stream is dying. So > appl

Re: How to recover from ConsumerRebalanceFailedException ?

HI Jun, Do you want me to request Jira ticket for feature a notification for new consumer API and old consumer feature that consumer stream is dying. So application can try to restart it programmatically. I understand this is due to network or zk cluster instability. Let me know if you have alt

Re: How to recover from ConsumerRebalanceFailedException ?

Hi Jun, ZK cluster are up and running. What is best way to programmatically recover and I would try to exponential recovery process which I am willing to implement.So do you think monitoring "ZkClient-EventThread *" thread

Re: How to recover from ConsumerRebalanceFailedException ?

Is your ZK service alive at that point? If not, you just need to monitor the ZK server properly. Thanks, Jun On Mon, Nov 17, 2014 at 2:30 PM, Bhavesh Mistry wrote: > Hi Kafka Team, > > > I get following exception due to ZK/Network issues intermittently. How do > I recover from consumer thread

Re: How to recover from ConsumerRebalanceFailedException ?

Hi Kakfa team, So just monitor "ZkClient-EventThread *" threads via ThreadInfo[] threads = ManagementFactory.getThreadMXBean().; and if this ZkClient-EventThread

How to recover from ConsumerRebalanceFailedException ?

Hi Kafka Team, I get following exception due to ZK/Network issues intermittently. How do I recover from consumer thread dying *programmatically* and restart source because we have alerts that due to this error we have partition OWNERSHIP is *none* ? Please let me know how to restart source and

Re: ConsumerRebalanceFailedException

rConsumerConnector.scala:326) > [06/02/14 08:59:18:018 AM EST] 215 INFO mkconsumer.MKConsumer: Put to > table Credit.SMC.MUNI info put gdm entity Credit.SMC.MUNI, key > MUNI.SMC.134103391 takes 4 ms > [06/02/14 08:59:18:018 AM EST] 215 INFO mkconsumer.MKConsumer

RE: ConsumerRebalanceFailedException

25, 2014 4:03 PM To: users@kafka.apache.org Subject: Re: ConsumerRebalanceFailedException Could you send around the consumer log when it throws ConsumerRebalanceFailedException. It should state the reason for the failed rebalance attempts. Thanks, Neha On Tue, Feb 25, 2014 at 12:01 PM, Yu, Li

Re: ConsumerRebalanceFailedException

Could you send around the consumer log when it throws ConsumerRebalanceFailedException. It should state the reason for the failed rebalance attempts. Thanks, Neha On Tue, Feb 25, 2014 at 12:01 PM, Yu, Libo wrote: > Hi all, > > I tried to reproduce this exception. In case one, when

ConsumerRebalanceFailedException

Hi all, I tried to reproduce this exception. In case one, when no broker was running, I launched all consumers and got this exception. In case two, while the consumers and brokers were running, I shutdown all brokers one by one and did not see this exception. I wonder why in case two this except

Re: ConsumerRebalanceFailedException

ice within second without any wait. > > *"connector.shutdown()" > > is good option for this but that will not work if consumer is killed > > abnormally using kill -9*. > > > > Other option i am seeing to put > "Thread.sleep(sessionTimeoutMilliseco

Re: ConsumerRebalanceFailedException

s killed > abnormally using kill -9*. > > Other option i am seeing to put "Thread.sleep(sessionTimeoutMilliseconds)" > in consumer service before start but that is also not good option. > > *When ConsumerRebalanceFailedException occurs then it stop consumes the > data. B

Re: ConsumerRebalanceFailedException

*. > > Other option i am seeing to put "Thread.sleep(sessionTimeoutMilliseconds)" > in consumer service before start but that is also not good option. > > *When ConsumerRebalanceFailedException occurs then it stop consumes the > data. But expected behaviour should be like this : If C**onsumer

Re: ConsumerRebalanceFailedException

ot;Thread.sleep(sessionTimeoutMilliseconds)" in consumer service before start but that is also not good option. *When ConsumerRebalanceFailedException occurs then it stop consumes the data. But expected behaviour should be like this : If C**onsumerRe* *balanceFai**ledExcepti**on occurs due to zooke

Re: ConsumerRebalanceFailedException

s, > > Libo > > > -Original Message- > From: Jun Rao [mailto:jun...@gmail.com] > Sent: Monday, December 30, 2013 11:13 AM > To: users@kafka.apache.org > Subject: Re: ConsumerRebalanceFailedException > > If the consumer is not shut down properly, it

RE: ConsumerRebalanceFailedException

To: users@kafka.apache.org Subject: Re: ConsumerRebalanceFailedException If the consumer is not shut down properly, it will take zookeeper.session.timeout.ms before the consumer is deregistered from ZK. If you restart the consumer before that, rebalances may fail. Make sure tha

Re: ConsumerRebalanceFailedException

er 5-6 sec then it started > working properly without throwing any exception. > > If i start consumer immediately after killing that then > ConsumerRebalanceFailedException occurs. > > Please help !! > > -- > *Thanks & Regards* > *Hanish Bansal* >

ConsumerRebalanceFailedException

0 If i kill the consumer and start that again after 5-6 sec then it started working properly without throwing any exception. If i start consumer immediately after killing that then ConsumerRebalanceFailedException occurs. Please help !! -- *Thanks & Regards* *Hanish Bansal*

Re: ConsumerRebalanceFailedException

n. How long is the interval between the retries? > Is it max.fetch.wait + socket.timeout.ms? > Thanks. > > Libo > > > -Original Message- > From: Jun Rao [mailto:jun...@gmail.com] > Sent: Monday, December 02, 2013 11:55 AM > To: users@kafka.apache.org > Subject

RE: ConsumerRebalanceFailedException

Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Monday, December 02, 2013 11:55 AM To: users@kafka.apache.org Subject: Re: ConsumerRebalanceFailedException Is the failure on the last rebalance? If so, some partitions will not have any consumers. A common reason for rebalance failure is that

Re: ConsumerRebalanceFailedException

: Friday, November 29, 2013 8:50 PM > To: users@kafka.apache.org > Subject: Re: ConsumerRebalanceFailedException > > Transient rebalance failures are ok. However, it's important that the last > rebalance in a sequence succeeds. Otherwise, some of the partitions will > not be consumed by

RE: ConsumerRebalanceFailedException

n Rao [mailto:jun...@gmail.com] Sent: Friday, November 29, 2013 8:50 PM To: users@kafka.apache.org Subject: Re: ConsumerRebalanceFailedException Transient rebalance failures are ok. However, it's important that the last rebalance in a sequence succeeds. Otherwise, some of the partitions w

Re: ConsumerRebalanceFailedException

x27;s > log. My question is whether users need to do > anything to handle ConsumerRebalanceFailedException. > > This is from consumer log: > > [28/11/13 16:38:56:056 PM EST] 102 ERROR > consumer.ZookeeperConsumerConnector: [xx > ], error during syncedRebalance >

RE: ConsumerRebalanceFailedException

n found in consumer's log. My question is whether users need to do anything to handle ConsumerRebalanceFailedException. This is from consumer log: [28/11/13 16:38:56:056 PM EST] 102 ERROR consumer.ZookeeperConsumerConnector: [xx ], error during

Re: ConsumerRebalanceFailedException

tion? > > Regards, > > Libo > > > -Original Message- > From: Florin Trofin [mailto:ftro...@adobe.com] > Sent: Tuesday, July 16, 2013 4:20 PM > To: users@kafka.apache.org > Subject: Re: ConsumerRebalanceFailedException > > Yes, I think these are two se

RE: ConsumerRebalanceFailedException

: ConsumerRebalanceFailedException Yes, I think these are two separate issues. F. On 7/16/13 11:32 AM, "Joel Koshy" wrote: >From a user's perspective, ConsumerRebalanceException is a bit cryptic >-I think the other thread was to provide a more informative message and >also b

Re: ConsumerRebalanceFailedException

Yes, I think these are two separate issues. F. On 7/16/13 11:32 AM, "Joel Koshy" wrote: >From a user's perspective, ConsumerRebalanceException is a bit cryptic >-I think the other thread was to provide a more informative message >and also be able to recover when a broker does come up (fixed in

Re: ConsumerRebalanceFailedException

>From a user's perspective, ConsumerRebalanceException is a bit cryptic -I think the other thread was to provide a more informative message and also be able to recover when a broker does come up (fixed in KAFKA-969). Thanks, Joel On Tue, Jul 16, 2013 at 11:04 AM, Vaibhav Puranik wrote: > Thank

Re: ConsumerRebalanceFailedException

Thank you Joel. In a different but related thread, somebody is asking to rename the exception as NoBrokerAvailableExcption. But given the description above, the exception seems to be named appropriately. Regards, Vaibhav On Tue, Jul 16, 2013 at 12:05 AM, Joel Koshy wrote: > Yes - rebalance =>

Re: ConsumerRebalanceFailedException

Yes - rebalance => consumers trying to coordinate through ZK. Rebalances can happen when one or more of the following happen: - a consumed topic partition appears or disappears - i.e., if a broker comes or goes. - a consumer instance in the group comes or goes "goes" could also be triggered by sess

Re: ConsumerRebalanceFailedException when broker unavailable

mer (written in scala) but don't launch a Kafka broker (I > am not using replication in my configuration). > > What happens in this particular scenario is that my consumer hits the > createMessageStreams which internally makes 4 failed attempts and then it

ConsumerRebalanceFailedException

Hi all, We have a small Kafka cluster (0.7.1 - 3 nodes) in EC2. The load is about 200 million events per day, each being few kilobytes. We have a single node zookeeper. Yesterday suddenly our Kafka clients started throwing the following exception: java.lang.RuntimeException: kafka.common.Consumer

ConsumerRebalanceFailedException when broker unavailable

ns in this particular scenario is that my consumer hits the createMessageStreams which internally makes 4 failed attempts and then it raises a ConsumerRebalanceFailedException: kafka.common.ConsumerRebalanceFailedException: zfilter_vm-florin-ubuntu-d-12-1373675347482-ff93b841 can't rebalance a

Re: ConsumerRebalanceFailedException

t; > I have a Scala Kafka consumer, written in the image of the Java code from > the Quickstart doc. I use the consumer for prototyping and debugging so I > start and stop it quite frequently. I noticed that every now and then, upon > starting the consumer, a ConsumerRebalanceFailedExcept

Re: ConsumerRebalanceFailedException

> Hi -- > >> > >> I have a Scala Kafka consumer, written in the image of the Java code > >>from > >> the Quickstart doc. I use the consumer for prototyping and debugging so > >>I > >> start and stop it quite frequently. I noticed that every now a

Re: ConsumerRebalanceFailedException

>> I have a Scala Kafka consumer, written in the image of the Java code >>from >> the Quickstart doc. I use the consumer for prototyping and debugging so >>I >> start and stop it quite frequently. I noticed that every now and then, >>upon >> starting t

Re: ConsumerRebalanceFailedException

oticed that every now and then, upon > starting the consumer, a ConsumerRebalanceFailedException is thrown (see > below). What's causing the problem this exception is signaling and what are > the recommended practices for avoiding it? I am using Kafka 0.7.2 that I > built with

ConsumerRebalanceFailedException

Hi -- I have a Scala Kafka consumer, written in the image of the Java code from the Quickstart doc. I use the consumer for prototyping and debugging so I start and stop it quite frequently. I noticed that every now and then, upon starting the consumer, a ConsumerRebalanceFailedException is