A cheaper approach is to use ephemeral disk and have the replicas across AZs. You can lose up to 2 AZs and still avoid data loss. You mitigate complete region failure using mirror maker to replicate to another region.
On Tue, Dec 3, 2013 at 8:41 AM, Philip O'Toole <phi...@loggly.com> wrote: > Yeah, we use provisioned IOPs EBS so we have good performance. > > We run 3 ZK nodes, one in each AZ. Quorum 2. > > Philip > > On Dec 3, 2013, at 7:31 AM, Timothy Chen <tnac...@gmail.com> wrote: > > > Hi Philip, > > > > So I wonder if you guys hit disk perf problems with EBS? It seems quite > common in the past but I haven't tried recently. > > > > Also can you share how you guys deployed zookeeper in AWS so that a > qurom is always available? > > > > Tim > > > > Sent from my iPhone > > > >> On Dec 2, 2013, at 5:15 PM, Steve Morin <steve.mo...@gmail.com> wrote: > >> > >> Philip this is definitely useful > >> > >>> On Dec 2, 2013, at 14:55, Surendranauth Hiraman < > suren.hira...@sociocast.com> wrote: > >>> > >>> S Ahmed, > >>> > >>> This combination of Kafka and Storm to process streaming data is > becoming > >>> pretty common. Definitely worth looking at. > >>> > >>> The throughput will vary depending on your workload (cpu usage, etc.) > and > >>> if you're talking to a backend, of course. But it scales very well. > >>> > >>> -Suren > >>> > >>> > >>> > >>>> On Mon, Dec 2, 2013 at 5:49 PM, S Ahmed <sahmed1...@gmail.com> wrote: > >>>> > >>>> Interesting. So twitter storm is used to basically process the > messages on > >>>> kafka? I'll have to read-up on storm b/c I always thought the use > case > >>>> was a bit different. > >>>> > >>>> > >>>> > >>>>> On Sun, Dec 1, 2013 at 9:59 PM, Joe Stein <joe.st...@stealth.ly> > wrote: > >>>>> > >>>>> Awesome Philip, thanks for sharing! > >>>>> > >>>>>> On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <phi...@loggly.com> > >>>>> wrote: > >>>>> > >>>>>> A couple of us here at Loggly recently spoke at AWS reinvent, on > how we > >>>>>> use Kafka 0.72 in our ingestion pipeline. The slides are at the link > >>>>> below, > >>>>>> and may be of interest to people on this list. > >>>> > http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013 > >>>>>> > >>>>>> Any questions, let me know, though I can't promise I can answer > >>>>>> everything. Can't give the complete game away. :-) > >>>>>> > >>>>>> As always, Kafka rocks! > >>>>>> > >>>>>> Philip > >>> > >>> > >>> > >>> -- > >>> ___________________________________________ > >>> Available at these partners: > >>> [image: CloudFlare | shopify | Bigcommerce] > >>> > >>> SUREN HIRAMAN, VP TECHNOLOGY > >>> SOCIOCAST > >>> Simple. Powerful. Predictions. > >>> > >>> 96 SPRING STREET, 7TH FLOOR > >>> NEW YORK, NY 10012 > >>> O: (917) 525-2466 ext. 105 > >>> F: 646.349.4063 > >>> E: suren.hira...@sociocast.com > >>> W: www.sociocast.com > >>> > >>> Increase Conversion Rates up to 500%. Go to www.sociocast.com and > enter > >>> your URL for a free trial! >
