kafka.network.RequestMetrics.FetchFollower metrics

[Ian Kallen]

I’m comparing some of the performance differences between two clusters that are 
theoretically identical except for the physical networks they’re on (configs 
and hardware are identical, each has a local zookeeper ensemble). I noticed 
that the slower cluster is exhibiting significantly higher times in the 
kafka.network.RequestMetrics.FetchFollower Mbean, notice the values are ~ 2x - 
3x higher from the “Cluster B” broker

Metric  Cluster A brokerCluster B broker
50thPercentile  30  106
75thPercentile  68  263
95thPercentile  161 539
98thPercentile  219 695
999thPercentile 459 1151
99thPercentile  289 777
Count   51591577518675
Max 10004   2554
Mean50  170
Min 0   0
StdDev  59  187

Are these times that brokers with follower partitions are taking to fetch 
updates from the leaders? If someone familiar with this metric can shed some 
light on what it’s measuring, I’d really appreciate it. I’d like to be able to 
characterize more specifically what this metric is measuring to better isolate 
the latencies (disk, network, zookeeper roundtrips, something else…? ).  If 
there’s a guide to what the specific MBeans are exposing, pointers to that 
would be fantastic.

Thanks,
-Ian


--
Ian Kallen / @spidaman
“Whatever comes out of these gates, we've got a better chance of survival if we 
work together.” - General Maximus

partitions not syncing or becoming leaders

[Ian Kallen]

One of the three brokers in a Kafka cluster (v0.8.2.2) is seems be missing ISRs 
and even the partitions that have ISRs on it won’t be elected leaders for those 
partitions. Here’s a topic describe output:

Topic:lia.stage.raw_events PartitionCount:20 ReplicationFactor:3 
Configs:retention.ms=25920
Topic: lia.stage.raw_events Partition: 0 Leader: 0 Replicas: 1,2,0 Isr: 0,1,2
Topic: lia.stage.raw_events Partition: 1 Leader: 2 Replicas: 2,0,1 Isr: 0,2,1
Topic: lia.stage.raw_events Partition: 2 Leader: 0 Replicas: 0,1,2 Isr: 0,2
Topic: lia.stage.raw_events Partition: 3 Leader: 0 Replicas: 1,0,2 Isr: 0,2
Topic: lia.stage.raw_events Partition: 4 Leader: 2 Replicas: 2,1,0 Isr: 0,1,2
Topic: lia.stage.raw_events Partition: 5 Leader: 0 Replicas: 0,2,1 Isr: 0,2
Topic: lia.stage.raw_events Partition: 6 Leader: 0 Replicas: 1,2,0 Isr: 0,1,2
Topic: lia.stage.raw_events Partition: 7 Leader: 2 Replicas: 2,0,1 Isr: 0,2,1
Topic: lia.stage.raw_events Partition: 8 Leader: 0 Replicas: 0,1,2 Isr: 0,2
Topic: lia.stage.raw_events Partition: 9 Leader: 0 Replicas: 1,0,2 Isr: 0,2
Topic: lia.stage.raw_events Partition: 10 Leader: 2 Replicas: 2,1,0 Isr: 0,1,2
Topic: lia.stage.raw_events Partition: 11 Leader: 0 Replicas: 0,2,1 Isr: 0,2
Topic: lia.stage.raw_events Partition: 12 Leader: 0 Replicas: 1,2,0 Isr: 0,1,2
Topic: lia.stage.raw_events Partition: 13 Leader: 2 Replicas: 2,0,1 Isr: 0,2,1
Topic: lia.stage.raw_events Partition: 14 Leader: 0 Replicas: 0,1,2 Isr: 0,2
Topic: lia.stage.raw_events Partition: 15 Leader: 0 Replicas: 1,0,2 Isr: 0,2
Topic: lia.stage.raw_events Partition: 16 Leader: 2 Replicas: 2,1,0 Isr: 0,1,2
Topic: lia.stage.raw_events Partition: 17 Leader: 0 Replicas: 0,2,1 Isr: 0,2
Topic: lia.stage.raw_events Partition: 18 Leader: 0 Replicas: 1,2,0 Isr: 0,1,2
Topic: lia.stage.raw_events Partition: 19 Leader: 2 Replicas: 2,0,1 Isr: 0,2,1

None of the brokers are logging any errors concerning replication (or any 
errors), but as you can see, broker “1” does not have ISR for a lot of 
partitions (only “0" or “2”) and of the partitions on “1”, running 
kafka-preferred-replica-election.sh doesn’t move leadership to any of them.  
How do I diagnose further why that instance isn’t syncing or electing partition 
leadership like the others in the cluster?
Thanks,
-Ian

Re: kafka metrics emitting to graphite

[Ian Kallen]

I plan on trying this out https://github.com/airbnb/kafka-statsd-metrics2 but 
yea, JMXTrans is a good option for getting mbean values into graphite as well.




On 10/11/15, 8:18 PM, "sunil kalva"  wrote:

>tx ben,
>i though we have native support for graphite in kafka, i will start
>exploring this.
>
>On Sun, Oct 11, 2015 at 7:00 PM, Ben Stopford  wrote:
>
>> Hi Sunil
>>
>> Try using JMXTrans to expose Kafka’s internal JMX metrics to graphite.
>>
>> https://github.com/jmxtrans/jmxtrans > >
>>
>> B
>> > On 11 Oct 2015, at 11:19, sunil kalva  wrote:
>> >
>> > How to configure, to emit kafka broker metrics to graphite.
>> >
>> > t
>> > SunilKalva
>>
>>