One of the three brokers in a Kafka cluster (v0.8.2.2) is seems be missing ISRs and even the partitions that have ISRs on it won’t be elected leaders for those partitions. Here’s a topic describe output:
Topic:lia.stage.raw_events PartitionCount:20 ReplicationFactor:3 Configs:retention.ms=259200000 Topic: lia.stage.raw_events Partition: 0 Leader: 0 Replicas: 1,2,0 Isr: 0,1,2 Topic: lia.stage.raw_events Partition: 1 Leader: 2 Replicas: 2,0,1 Isr: 0,2,1 Topic: lia.stage.raw_events Partition: 2 Leader: 0 Replicas: 0,1,2 Isr: 0,2 Topic: lia.stage.raw_events Partition: 3 Leader: 0 Replicas: 1,0,2 Isr: 0,2 Topic: lia.stage.raw_events Partition: 4 Leader: 2 Replicas: 2,1,0 Isr: 0,1,2 Topic: lia.stage.raw_events Partition: 5 Leader: 0 Replicas: 0,2,1 Isr: 0,2 Topic: lia.stage.raw_events Partition: 6 Leader: 0 Replicas: 1,2,0 Isr: 0,1,2 Topic: lia.stage.raw_events Partition: 7 Leader: 2 Replicas: 2,0,1 Isr: 0,2,1 Topic: lia.stage.raw_events Partition: 8 Leader: 0 Replicas: 0,1,2 Isr: 0,2 Topic: lia.stage.raw_events Partition: 9 Leader: 0 Replicas: 1,0,2 Isr: 0,2 Topic: lia.stage.raw_events Partition: 10 Leader: 2 Replicas: 2,1,0 Isr: 0,1,2 Topic: lia.stage.raw_events Partition: 11 Leader: 0 Replicas: 0,2,1 Isr: 0,2 Topic: lia.stage.raw_events Partition: 12 Leader: 0 Replicas: 1,2,0 Isr: 0,1,2 Topic: lia.stage.raw_events Partition: 13 Leader: 2 Replicas: 2,0,1 Isr: 0,2,1 Topic: lia.stage.raw_events Partition: 14 Leader: 0 Replicas: 0,1,2 Isr: 0,2 Topic: lia.stage.raw_events Partition: 15 Leader: 0 Replicas: 1,0,2 Isr: 0,2 Topic: lia.stage.raw_events Partition: 16 Leader: 2 Replicas: 2,1,0 Isr: 0,1,2 Topic: lia.stage.raw_events Partition: 17 Leader: 0 Replicas: 0,2,1 Isr: 0,2 Topic: lia.stage.raw_events Partition: 18 Leader: 0 Replicas: 1,2,0 Isr: 0,1,2 Topic: lia.stage.raw_events Partition: 19 Leader: 2 Replicas: 2,0,1 Isr: 0,2,1 None of the brokers are logging any errors concerning replication (or any errors), but as you can see, broker “1” does not have ISR for a lot of partitions (only “0" or “2”) and of the partitions on “1”, running kafka-preferred-replica-election.sh doesn’t move leadership to any of them. How do I diagnose further why that instance isn’t syncing or electing partition leadership like the others in the cluster? Thanks, -Ian
