In my experience, anything smaller than m1.xlarge isn't really suitable for I/O intensive high performance stuff. I would guess that, for Kafka, a single m1.xlarge would outperform two m1.large. I have no hard evidence to support this however.
What I'd like to see are some benchmarks comparing 12 m1.large to 6 m1.xlarge to 1 hi1.4xlarge. Another interesting note is with the m1.xlarge you can get "optimized" EBS instances with a claimed 1000 Mbps I/O throughput. On Nov 19, 2012, at 7:36 PM, Bae, Jae Hyeon wrote: > Yes, 12 m1.large instances couldn't handle more than 12k messages per > second in our environment. When the traffic goes up to 12k/sec, kafka > clusters started to be throttling. I am not sure how much one > m1.xlarge will outperform 2 m1.large instances because m1.xlarge is > twice expensive than m1.large. I vote that two m1.large will be better > than 1 m1.xlarge. > > Speaking of EBS volume, EBS writing performance is not good and it's > expensive. I hope replication in 0.8 will save us. > > On Sun, Nov 18, 2012 at 3:36 PM, Neha Narkhede <neha.narkh...@gmail.com> > wrote: >>>> Single m1.large instance's capacity was 10k/sec. >> >> When you say capacity, did you mean the I/O or network capacity on the >> m1.large instances ? >> >> Thanks, >> Neha >> >> >> On Sun, Nov 18, 2012 at 3:26 PM, Bae, Jae Hyeon <metac...@gmail.com> wrote: >>> I am running kafka on ec2 with m1.large instance. I think that large >>> number of low end servers will outperform small number of high end >>> servers, but I am not sure I am correct. >>> >>> I assumed 12 m1.large instances would be able to handle more than 6 >>> billion rows in a day but my expectation was wrong. Single m1.large >>> instance's capacity was 10k/sec. If we want to have a room, we'd >>> better assume kafka on m1.large maximum capacity is 8k/sec. >>> >>> On Fri, Nov 16, 2012 at 5:57 PM, Joel Koshy <jjkosh...@gmail.com> wrote: >>>> At least based on prior threads (discussing experiences/issues with ec2), >>>> there should be a number of people on this list who can help you. It would >>>> be helpful if we have a ec2-operations wiki @ >>>> https://cwiki.apache.org/confluence/display/KAFKA/Index . Would people be >>>> interested in sharing operational experiences there? I'm thinking something >>>> similar to the operations wiki that's already available ( >>>> https://cwiki.apache.org/confluence/display/KAFKA/Operations). >>>> >>>> Joel >>>> >>>> >>>> On Fri, Nov 16, 2012 at 5:04 PM, Senthilvel Rangaswamy >>>> <senthil...@gmail.com >>>>> wrote: >>>> >>>>> Have folks implemented large installations on Kafka on Amazon EC2. I am >>>>> looking for best practices. Like the kind of nodes, EBS vs Instance store >>>>> etc., >>>>> >>>>> -- >>>>> ..Senthil >>>>> >>>>> "If there's anything more important than my ego around, I want it >>>>> caught and shot now." >>>>> - Douglas Adams. >>>>>
