Evan, Thanks for sharing this. Not too sure about the difference btw EBS and local disk. EBS could be caching writes in non-volatile memory. Therefore, you only pay network cost, but not disk I/O cost.
We have fixed the reporting problem in performance test in 0.7. I recommend that you try the 0.7 release. Jun On Thu, Feb 2, 2012 at 4:04 PM, Evan Chan <e...@ooyala.com> wrote: > Hi folks, > > I did a performance test on EC2 using one Kafka box on an m1.large, and > separate m1.large boxes for the producer-perf-shell and the > consumer-perf-shell. > I'm using Kafka 0.6 with no mirroring. > With the message size set to 1000 bytes, here is what I found: > > - Writing Kafka messages to /mnt (EBS), we can write 21,190 messages per > second. > - If we write Kafka messages to /tmp (local disk), it's only slightly > faster at 21,944 messages / sec > - The number of partitions doesn't seem to have a bearing on > write/consumer performance > > It's pretty interesting that writing to /mnt or EBS doesn't seem to really > hurt the bandwidth. > > I also found that the kafka-consumer-perf-test seems to misreport the > "nMsgs/sec". It goes up from a small number at the beginning of the test > to a large number, which tells me that there is some elapsed / original > timestamp that was set waaay before the first message was consumed. Maybe > this is fixed in 0.7? > > I'd be interested to compare notes with others who have deployed Kafka to > EC2. > > thanks, > Evan > -- > -- > *Evan Chan* > Senior Software Engineer | > e...@ooyala.com | (650) 996-4600 > www.ooyala.com | blog <http://www.ooyala.com/blog> | > @ooyala<http://www.twitter.com/ooyala> >
