Hi All, Our Kafka cluster ran out of disk space yesterday. After we freed up the cluster disk space and adjusted the broker data retention policy, we noticed that the cluster partition was not balanced based on topic describe script came from Kafka 0.8.1.1 distribution. So, we tried to rebalance the partition using the kafka-reassign-partitions.sh. After sometime later, we ran out of disk space on 2 brokers in the cluster while the rest have plenty of disk space left.
This seems to suggest that only two brokers were receiving messages. We have not changed the broker partition from our producer which uses a random partition key strategy. String uuid = UUID.randomUUID().toString(); KeyedMessage<String, String> data = new KeyedMessage<String, String>( "myKafkaTopic" uuid, msgBuilder.toString()); Questions 1. Is partition reassignment required after disk full or when some of the brokers are not healthy? 2. Is there a broker config that we can use to auto rebalance the broker partition? Should "auto.leader.rebalance.enable" set to true? 2. How do we recover from situation like this? We pretty much use default configuration on the broker. Thanks, Connie
