Stephane Maarek created KAFKA-5851: -------------------------------------- Summary: CPU overhead to having partitions (even inactive) Key: KAFKA-5851 URL: https://issues.apache.org/jira/browse/KAFKA-5851 Project: Kafka Issue Type: Bug Environment: 0.11.0.0, 0.10.x Reporter: Stephane Maarek
We're running on three r4.xlarge, and this is our dev environment. Some people managed to create 1200 topics with 3 partitions, so we end up at 4000 partitions per broker (we have a replication factor of 3). Even though no data goes through the cluster (it sits at a comfortable 20 messages per seconds), we saw CPU at 100% (out of 400%). I went ahead today and deleted 700 topics that I knew were unused. And CPU went drastically down. See image !https://i.imgur.com/OIPTwDM.png! We use the defaults for our brokers, and they use PLAINTEXT internally to replicate. I'm not sure of the root cause of this, threads, replication, log cleanup, etc, and I guess it wouldn't be too hard to replicate (just create 1000 topics on a vanilla cluster and see CPU go up). Hope that helps -- This message was sent by Atlassian JIRA (v6.4.14#64029)