I noticed similar behavior on similar small 3 Kafka broker with 3 ZooKeeper node cluster, Kafka 0.8.1.1 and ZooKeeper 3.4.6, with ~5K topics, most of them with single partition, replication factor of 1, and most of them unused for long time, but brokers are busy and performance especially producer performance is slim. Plan to investigate root cause. Would be nice if it was documented, by design how many topics/partitions can be handled by single Kafka broker node, and then also scalability limits/constraints. Both Kafka and ZooKeeper need to be checked for limits.
Kind regards, Stevo Slavic. On Wed, May 6, 2015 at 1:32 PM, Pavel Sýkora <pav...@seznam.cz> wrote: > Hi, > > > > I've started playing with Kafka_2.10-0.8.2.1 (a cluster of 3 VM nodes). I > found a strange behaviour: Inbound throughput was about 60 thousand > messages > per second for a partition with 50 partitions. But after adding topics with > 950 partitions total, the performance dropped under 25 thousand messages > per > second, even if the 950 new partitions were unused and empty. In other > words: a simple existence of empty unused partitions can drop Kafka > performance to less than 50 %. > > > > > Has anybody else observed such Kafka behaviour? Does anybody have an > explanation for this? > > > > > Regards, > > > > > Pavel >
