If you raid EBS volumes you're pretty much protected from data loss as an added benefit. So yeah if a broker goes down, you don't have the data until you bring it back up, but you CAN bring it back up, even if the box dies.
On Mon, Nov 19, 2012 at 3:34 PM, Matt Jones <m...@pinterest.com> wrote: > We have pretty much the same setup as this, running on m1.large instances. > > On Mon, Nov 19, 2012 at 11:44 AM, Matthew Rathbone > <matt...@foursquare.com>wrote: > > > We RAID-0 4 EBS disks, we find that to be most performant, although it > does > > leave you more vulnerable to EBS network errors and outages. Ideally, if > > you're spread across AZ's you could do some clever routing to > > geographically local brokers, whilst keeping the others as a backup in > case > > of failure. > > > > In the same vein, we have N+1 brokers, where N is how many we reasonably > > think we need, this way we can hopefully survive outages. > > > > > > On Mon, Nov 19, 2012 at 1:30 PM, David Arthur <mum...@gmail.com> wrote: > > > > > I'd only consider m1.xlarge and higher for Kafka. The m1.xlarge have > > > "high" I/O performance according to Amazon. This is disk I/O and > network > > > I/O performance. Of course you need to use EBS volumes if you want your > > > Kafka brokers to survive reboots - you can expect reboots on AWS. Some > > > people have reported I/O improvements by RAIDing EBS volumes ( > > > http://alestic.com/2009/06/ec2-ebs-raid). Deploying in the same region > > as > > > your application will also improve performance. > > > > > > > > > On Nov 16, 2012, at 8:04 PM, Senthilvel Rangaswamy wrote: > > > > > > > Have folks implemented large installations on Kafka on Amazon EC2. I > am > > > > looking for best practices. Like the kind of nodes, EBS vs Instance > > store > > > > etc., > > > > > > > > -- > > > > ..Senthil > > > > > > > > "If there's anything more important than my ego around, I want it > > > > caught and shot now." > > > > - Douglas Adams. > > > > > > > > > > > > -- > > Matthew Rathbone > > Foursquare | Software Engineer | Server Engineering Team > > matt...@foursquare.com | @rathboma <http://twitter.com/rathboma> | > > 4sq<http://foursquare.com/rathboma> > > > -- Matthew Rathbone Foursquare | Software Engineer | Server Engineering Team matt...@foursquare.com | @rathboma <http://twitter.com/rathboma> | 4sq<http://foursquare.com/rathboma>
