Re: under replicated topics

Brokers may have temporary problems catching up with the leaders. So I would not worry about it if it happens only once a while and goes away. Occasionally we have seen under replicated topics for long time, which might be caused by ZooKeeper session problem as indicated by such log messages:

RE: under replicated topics

We restarted the Kafka brokers this morning. That fixed the issue. Gene Robichaux Manager, Database Operations Match.com 8300 Douglas Avenue I Suite 800 I Dallas, TX  75225 -Original Message- From: Allen Wang [mailto:aw...@netflix.com.INVALID] Sent: Tuesday, December 30, 2014 1:38 PM

Re: Trying to figure out kafka latency issues

The produce to the local log is the time taken to write to the local filesystem. If you see that spike up then that is the culprit. What I was referring to is the line in kafka-request.log that looks something like: Completed request:%s from client

Re: under replicated topics

We have seen cases with 0.8.1 when, under load, replication threads would hang up and not transfer data any longer. Restarting clears this.  I haven't found a way to monitor for this in a nice way, other than seeing partitions stay under-replicated for long periods of time. Sent from my 

Re: under replicated topics

I am still trying to find a way to detect how far behind a replica is, nicely, so I can differentiate between 10 offsets and 1 offsets behind.  ‎ This would help with problems like this one, as we often have replicas that are just slightly behind, due to Bursty traffic, but the ones

Re: under replicated topics

It sounds like you are looking for FetcherLagMetrics. See https://kafka.apache.org/documentation.html#monitoring you can (if you find your ISR shrink/growth rate flapping) increase the max lag setting replica.lag.max.messages (default is 4000) depending on where you see this lag often hovering at

Kafka JMX metrics is not a registered bean

I saw the same issue, here's the solution... escape the double quotes... java -d64 -Xmx64M -jar ./jmxcmd.jar - 192.168.2.222:9005 \kafka.server\:type=\BrokerTopicMetrics\,name=\AllTopicsMessagesInPerSec\ Count RateUnit FiveMinuteRate FifteenMinuteRate OneMinuteRate MeanRate Wrapping with

Re: Trying to figure out kafka latency issues

Got it. I had to enable to see the logs. Here are two files with the timings I got from the period I had logging enabled: Producer Requests : https://docs.google.com/spreadsheets/d/1spVFWsf7T8ZmwM0JMYnkBoyYufRhHav50sueFA9ztP8/edit?usp=sharing Fetch Requests:

Re: Trying to figure out kafka latency issues

Edit: I had to set kafka.request.logger=TRACE to see the request timings. On Tue, Dec 30, 2014 at 4:37 PM, Rajiv Kurian ra...@signalfuse.com wrote: Got it. I had to enable to see the logs. Here are two files with the timings I got from the period I had logging enabled: Producer Requests :

Re: Trying to figure out kafka latency issues

Yes, the fetch request blocks until there is data for you, so you should see high remote time there. For the producer you should only see remote time if you have acks=-1. This represents the time it takes the message to replicate to other in-sync replicas. I agree if there are no slow requests

Re: Trying to figure out kafka latency issues

My broker logs say acks = 1 unless this is some other ack: Completed request:Name: ProducerRequest; Version: 0; CorrelationId: 858424; ClientId: ; *RequiredAcks: 1*; AckTimeoutMs: 3 ms; TopicAndPartition: I'll try to get a stand alone test going, and try to narrow down the problem. Thanks

Re: Kafka JMX metrics is not a registered bean

Oh, thanks. Both solutions work. 2014-12-31 7:03 GMT+08:00 Chris Cordray ccord...@sciencelogic.com: I saw the same issue, here's the solution... escape the double quotes... java -d64 -Xmx64M -jar ./jmxcmd.jar - 192.168.2.222:9005