Hi folks, I did a performance test on EC2 using one Kafka box on an m1.large, and separate m1.large boxes for the producer-perf-shell and the consumer-perf-shell. I'm using Kafka 0.6 with no mirroring. With the message size set to 1000 bytes, here is what I found:
- Writing Kafka messages to /mnt (EBS), we can write 21,190 messages per second. - If we write Kafka messages to /tmp (local disk), it's only slightly faster at 21,944 messages / sec - The number of partitions doesn't seem to have a bearing on write/consumer performance It's pretty interesting that writing to /mnt or EBS doesn't seem to really hurt the bandwidth. I also found that the kafka-consumer-perf-test seems to misreport the "nMsgs/sec". It goes up from a small number at the beginning of the test to a large number, which tells me that there is some elapsed / original timestamp that was set waaay before the first message was consumed. Maybe this is fixed in 0.7? I'd be interested to compare notes with others who have deployed Kafka to EC2. thanks, Evan -- -- *Evan Chan* Senior Software Engineer | e...@ooyala.com | (650) 996-4600 www.ooyala.com | blog <http://www.ooyala.com/blog> | @ooyala<http://www.twitter.com/ooyala>
