RE: steps path to kafka mastery

You can certainly try out things in a development environment. You could also think of contributing to the project itself as that will encourage you to develop deeper skills. But you are right. Kafka is pretty straightforward to setup and get running. The real skill IMO is in the ability to

Re: Dealing with large messages

In addition to the config changes mentioned in that post, you may also have to change producer config if you are using the new producer. Specifically, *max.request.size* and *request.timeout.ms * have to be increased to allow the producer to send large messages. On 6

Re: Tools/recommendations to debug performance issues?

Have you checked the consumer lag? You can use the offset checker tool to see if there is a lag. On 14 Sep 2015 18:36, "noah" wrote: > We're using 0.8.2.1 processing maybe 1 million messages per hour. Each > message includes tracking information with a timestamp for when it

Re: Question regarding to reconnect.backoff.ms

We did notice something similar. When a broker node (out of 3) went down, metadata calls continued to go to the failed node and producer kept failing. We were able to make it work by increasing the reconnect.backoff.ms to 1 second. Something similar was discussed earlier -

Re: Kafka metadata

Alternatively you can get the same metadata from Zookeeper If this is true, why does the producer API makes it necessary to supply a value for metadata.broker.list? I noticed that this wasn't the case in 0.7. On 8 Aug 2015 04:06, Lukas Steiblys lu...@doubledutch.me wrote: Hi Qi, Yes,

Re: How to monitor Kafka offset

Burrow works only if you are storing the offsets in kafka topic, not zookeeper. You can also take a look at Kafka web console ( it has a memory leak bug but a patch is available ). On 10 Jul 2015 09:34, Jiangjie Qin j...@linkedin.com.invalid wrote: You can take a look at Burrow. We use it in

Re: New producer: metadata update problem on 2 Node cluster.

suppose the problem went off because the dead node in your case might have came back up and allowed for a metadata update. Can you confirm this? Thanks, Mayuresh On Tue, May 5, 2015 at 5:10 AM, Rahul Jain rahul...@gmail.com wrote: We observed the exact

Re: New producer: metadata update problem on 2 Node cluster.

you're working with. -Ewen On Thu, May 7, 2015 at 12:06 AM, Rahul Jain rahul...@gmail.com wrote: Creating a new consumer instance *does not* solve this problem. Attaching the producer/consumer code that I used for testing. On Wed, May 6, 2015 at 6:31 AM, Ewen Cheslack-Postava e

Re: New producer: metadata update problem on 2 Node cluster.

We observed the exact same error. Not very clear about the root cause although it appears to be related to leastLoadedNode implementation. Interestingly, the problem went away by increasing the value of reconnect.backoff.ms to 1000ms. On 29 Apr 2015 00:32, Ewen Cheslack-Postava e...@confluent.io

Re: New producer: metadata update problem on 2 Node cluster.

back up and allowed for a metadata update. Can you confirm this? Thanks, Mayuresh On Tue, May 5, 2015 at 5:10 AM, Rahul Jain rahul...@gmail.com wrote: We observed the exact same error. Not very clear about the root cause although it appears to be related to leastLoadedNode