> Leadership moves automatically for at least a few of the topics, which > never happens when we run them on our prod, non-AWS hardware. This causes
Under normal operation (i.e., without broker failures) leadership should not move. Leader changes occur when brokers fail - due to GC, controlled shutdowns/bounces, or network partitioning. Do you see any "Broker failure callback" message on the controller log? Can you also check your broker logs to see if you see any zookeeper session expirations? Joel On Fri, Aug 01, 2014 at 04:01:59PM -0600, Marcos Juarez wrote: > Hi, > > We have a Kafka 0.8 cluster in a test environment (in this case, on AWS EC2 > nodes). Even though we've tried to run very little load on this test > cluster, it seems like the instances can't even keep up with that. > Leadership moves automatically for at least a few of the topics, which > never happens when we run them on our prod, non-AWS hardware. This causes > us to eventually have to rebalance the topics on those test clusters, which > is annoying. > > Can any of you point me to the set of conditions/thresholds that have to be > met for the Kafka cluster to decide to automatically move leadership of a > topic/partition to another replica in the ISR? I'd like to understand how > exactly Kafka does this, to see if we can provision an instance type for > those test Kakfa clusters that can handle the load without moving > leadership around. > > Thanks, > > Marcos Juarez
