Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
; Sent: Wednesday, August 31, 2016 10:56 PM > To: us...@kafka.apache.org > Cc: dev@kafka.apache.org > Subject: Re: Kafka consumers unable to process message > > The exceptions show one of the replica fetcher threads on the broker > failing which makes perfect sense since some of th

RE: Kafka consumers unable to process message

2016-08-31 Thread Ghosh, Achintya (Contractor)
t works. Thanks for your response. -Original Message- From: Jason Gustafson [mailto:ja...@confluent.io] Sent: Wednesday, August 31, 2016 10:56 PM To: us...@kafka.apache.org Cc: dev@kafka.apache.org Subject: Re: Kafka consumers unable to process message The exceptions show one of the replica

Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
16-09-01 02:23:04,926] INFO [Group Metadata Manager on Broker 4]: > Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator. > GroupMetadataManager) > [2016-09-01 02:33:04,926] INFO [Group Metadata Manager on Broker 4]: > Removed 0 expired offsets in 1 milliseconds. (kafka.coordinat

RE: Kafka consumers unable to process message

2016-08-31 Thread Ghosh, Achintya (Contractor)
riginal Message- From: Jason Gustafson [mailto:ja...@confluent.io] Sent: Wednesday, August 31, 2016 10:26 PM To: us...@kafka.apache.org Cc: dev@kafka.apache.org Subject: Re: Kafka consumers unable to process message Hi Achintya, Just to clarify, you did not take down either of the zookee

Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
props.put("key.deserializer", "org.apache.kafka.common. > serialization.StringDeserializer"); > props.put("value.deserializer", > "com.comcast.ps.kafka.object.CustomMessageDeSer"); > > Is it because of consumer is not able to get the b

RE: Kafka consumers unable to process message

2016-08-31 Thread Ghosh, Achintya (Contractor)
props.put("value.deserializer", "com.comcast.ps.kafka.object.CustomMessageDeSer"); Is it because of consumer is not able to get the broker metadata if it is trying to connect other datacenter's zookeeper server? I tried with to increate the zookeeper session timeout and connecti

Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
Kafka clients have tended to make broker retries transparent to the user. There's been discussion on various JIRAs about what we should do when all the known brokers become unreachable. One option is to revert to the configured bootstrap broker list, which is nice if you've configured a vip for boo

Re: Kafka consumers unable to process message

2016-08-31 Thread Jim Jagielski
Yeah, let's figure out the "best" action to take... Looks like something I'd like to get a handle on. > On Aug 31, 2016, at 4:05 PM, Jason Gustafson wrote: > > Hi Achintya, > > We have a JIRA for this problem: https://issues. > apache.org/jira/browse/KAFKA-3834. Do you expect the client to rai

Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
Hi Achintya, We have a JIRA for this problem: https://issues. apache.org/jira/browse/KAFKA-3834. Do you expect the client to raise an exception in this case or do you just want to keep it from blocking indefinitely? If the latter, you could escape the poll from another thread using wakeup(). Than

Kafka consumers unable to process message

2016-08-31 Thread Ghosh, Achintya (Contractor)
Hi there, Kafka consumer gets stuck at consumer.poll() method if my current datacenter is down and replicated messages are in remote datacenter. How to solve that issue? Thanks Achintya