Hello, Looks like we are hitting leader election bug. I've stopped one broker (104224873) on other brokers I see following:
WARN kafka.controller.ControllerChannelManager - [Channel manager on controller 104224863]: Not sending request Name: StopReplicaRequest; Version: 0; CorrelationId: 843100; ClientId: ; DeletePartitions: false; ControllerId: 104224863; ControllerEpoch: 8; Partitions: [mp-auth,169] to broker 104224873, since it is offline. Also describing topics returns this: Topic: mp-unknown Partition: 597 Leader: 104224873 Replicas: 104224874,104224873,104224875 Isr: 104224873,104224875 broker 104224873 is shut down, but it's still leader for the partition (at least for a couple of hours as I monitor it). Zookeeper cluster is healthy. ls /brokers/ids [104224874, 104224875, 104224863, 104224864, 104224871, 104224867, 104224868, 104224865, 104224866, 104224876, 104224877, 104224869, 104224878, 104224879] That broker is not registered in ZK.
