We upgraded a 9 broker cluster from version 0.8.1 to version 0.8.2.1. Actually we cherry-picked the commit at 41ba26273b497e4cbcc947c742ff6831b7320152 to get zkClient 0.5 because we ran into a bug described at https://issues.apache.org/jira/browse/KAFKA-824
Right after the update the CPU spiked quite a bit but I am guessing that is because the brokers were pulling in log segments from other brokers right after restart. The CPU remained elevated for a while and I thought it would come down after things settled down but the CPU has remained higher even after a day. Our steady state CPU on the brokers went from about 28% (0.8.1) to 34% (0.8.2.1). We do not use compression on any topic or partition. Our incoming traffic (number of messages/sec) has not increased at all. Our incoming bytes/sec has actually decreased because we managed to reduce the size of one our message types from 256 bytes to 32 bytes. The message size change was made hours after the Kafka version update and didn't seem to harm or help the cpu. The bytes-in/sec and bytes-out/sec metrics have definitely gone down after the message size reduction. Here is a link to the graph showing how the CPU went up - http://i.imgur.com/KVJLzsX.png?1 The restarts were done from 18:00 to 19:00 and I'd expect the CPU to go up at that time but I can't explain the steady state CPU rise. Are there any known performance regressions after 0.8.1? Any hints on what I should investigate if you think that this is not normal? Thanks, Rajiv
