We use m1.large's with ephemeral storage and get 20MB/sec using Kafka's built in benchmarking tool. No compression.
On Tue, Nov 20, 2012 at 7:52 AM, David Arthur <mum...@gmail.com> wrote: > In my experience, anything smaller than m1.xlarge isn't really suitable > for I/O intensive high performance stuff. I would guess that, for Kafka, a > single m1.xlarge would outperform two m1.large. I have no hard evidence to > support this however. > > What I'd like to see are some benchmarks comparing 12 m1.large to 6 > m1.xlarge to 1 hi1.4xlarge. > > Another interesting note is with the m1.xlarge you can get "optimized" EBS > instances with a claimed 1000 Mbps I/O throughput. > > On Nov 19, 2012, at 7:36 PM, Bae, Jae Hyeon wrote: > > > Yes, 12 m1.large instances couldn't handle more than 12k messages per > > second in our environment. When the traffic goes up to 12k/sec, kafka > > clusters started to be throttling. I am not sure how much one > > m1.xlarge will outperform 2 m1.large instances because m1.xlarge is > > twice expensive than m1.large. I vote that two m1.large will be better > > than 1 m1.xlarge. > > > > Speaking of EBS volume, EBS writing performance is not good and it's > > expensive. I hope replication in 0.8 will save us. > > > > On Sun, Nov 18, 2012 at 3:36 PM, Neha Narkhede <neha.narkh...@gmail.com> > wrote: > >>>> Single m1.large instance's capacity was 10k/sec. > >> > >> When you say capacity, did you mean the I/O or network capacity on the > >> m1.large instances ? > >> > >> Thanks, > >> Neha > >> > >> > >> On Sun, Nov 18, 2012 at 3:26 PM, Bae, Jae Hyeon <metac...@gmail.com> > wrote: > >>> I am running kafka on ec2 with m1.large instance. I think that large > >>> number of low end servers will outperform small number of high end > >>> servers, but I am not sure I am correct. > >>> > >>> I assumed 12 m1.large instances would be able to handle more than 6 > >>> billion rows in a day but my expectation was wrong. Single m1.large > >>> instance's capacity was 10k/sec. If we want to have a room, we'd > >>> better assume kafka on m1.large maximum capacity is 8k/sec. > >>> > >>> On Fri, Nov 16, 2012 at 5:57 PM, Joel Koshy <jjkosh...@gmail.com> > wrote: > >>>> At least based on prior threads (discussing experiences/issues with > ec2), > >>>> there should be a number of people on this list who can help you. It > would > >>>> be helpful if we have a ec2-operations wiki @ > >>>> https://cwiki.apache.org/confluence/display/KAFKA/Index . Would > people be > >>>> interested in sharing operational experiences there? I'm thinking > something > >>>> similar to the operations wiki that's already available ( > >>>> https://cwiki.apache.org/confluence/display/KAFKA/Operations). > >>>> > >>>> Joel > >>>> > >>>> > >>>> On Fri, Nov 16, 2012 at 5:04 PM, Senthilvel Rangaswamy < > senthil...@gmail.com > >>>>> wrote: > >>>> > >>>>> Have folks implemented large installations on Kafka on Amazon EC2. I > am > >>>>> looking for best practices. Like the kind of nodes, EBS vs Instance > store > >>>>> etc., > >>>>> > >>>>> -- > >>>>> ..Senthil > >>>>> > >>>>> "If there's anything more important than my ego around, I want it > >>>>> caught and shot now." > >>>>> - Douglas Adams. > >>>>> > > -- -- *Evan Chan* Senior Software Engineer | e...@ooyala.com | (650) 996-4600 www.ooyala.com | blog <http://www.ooyala.com/blog> | @ooyala<http://www.twitter.com/ooyala>
