Yes, 12 m1.large instances couldn't handle more than 12k messages per second in our environment. When the traffic goes up to 12k/sec, kafka clusters started to be throttling. I am not sure how much one m1.xlarge will outperform 2 m1.large instances because m1.xlarge is twice expensive than m1.large. I vote that two m1.large will be better than 1 m1.xlarge.
Speaking of EBS volume, EBS writing performance is not good and it's expensive. I hope replication in 0.8 will save us. On Sun, Nov 18, 2012 at 3:36 PM, Neha Narkhede <neha.narkh...@gmail.com> wrote: >>> Single m1.large instance's capacity was 10k/sec. > > When you say capacity, did you mean the I/O or network capacity on the > m1.large instances ? > > Thanks, > Neha > > > On Sun, Nov 18, 2012 at 3:26 PM, Bae, Jae Hyeon <metac...@gmail.com> wrote: >> I am running kafka on ec2 with m1.large instance. I think that large >> number of low end servers will outperform small number of high end >> servers, but I am not sure I am correct. >> >> I assumed 12 m1.large instances would be able to handle more than 6 >> billion rows in a day but my expectation was wrong. Single m1.large >> instance's capacity was 10k/sec. If we want to have a room, we'd >> better assume kafka on m1.large maximum capacity is 8k/sec. >> >> On Fri, Nov 16, 2012 at 5:57 PM, Joel Koshy <jjkosh...@gmail.com> wrote: >>> At least based on prior threads (discussing experiences/issues with ec2), >>> there should be a number of people on this list who can help you. It would >>> be helpful if we have a ec2-operations wiki @ >>> https://cwiki.apache.org/confluence/display/KAFKA/Index . Would people be >>> interested in sharing operational experiences there? I'm thinking something >>> similar to the operations wiki that's already available ( >>> https://cwiki.apache.org/confluence/display/KAFKA/Operations). >>> >>> Joel >>> >>> >>> On Fri, Nov 16, 2012 at 5:04 PM, Senthilvel Rangaswamy <senthil...@gmail.com >>>> wrote: >>> >>>> Have folks implemented large installations on Kafka on Amazon EC2. I am >>>> looking for best practices. Like the kind of nodes, EBS vs Instance store >>>> etc., >>>> >>>> -- >>>> ..Senthil >>>> >>>> "If there's anything more important than my ego around, I want it >>>> caught and shot now." >>>> - Douglas Adams. >>>>
