Nelz,
Great! Thanks for your reply. So it looks like your
architecture replicates data in both the "near" and "far"
groups. It looks like your architecture also ensures reads are
performed based on data locality.
Have you implemented some sort of quorum protocol to ensure
consistency for reads and writes?
I guess we will have have to do some similar stuff and ensure that
reads happen from the closest region.
Any suggestions are welcome.
Regards
Ganesh
On Jul 5, 8:12 pm, Nelz <nelz9...@gmail.com> wrote:
> We are doing some multi-datacenter stuff like this, but it takes more
> architecture than just adding more servers to the clusters.
>
> Basically, we have two different logical memcached setups. From each
> DC, we'd have a 'near' group and a 'far' group. The 'near' group
> handles the read-through cases and also gets data pushed into it,
> whereas the 'far' group only gets data pushed to it (not serving
> read-through requests) asynchronously.
>
> I don't think anyone has broken the physics barriers for deploying a
> fully consistent, fully distributed, fully consistent, zero-latency
> data store. ;-)
>
> - Nelz
>
>
>
>
>
>
>
> On Tue, Jul 5, 2011 at 03:06, TV Ganesh <tvganesh...@gmail.com> wrote:
> > Dustin,
> > Thanks for your response. I do agree with you that I am
> > essentially limited by the speed of light.
> > We intend to have the application across Amazon EC2 regions for two
> > reasons a) ensuring against
> > the outages like the one this May 2011 b) to provide a better
> > performance for US-West and US-East.
>
> > Regarding traffic originations from East & West we have convenient
> > geographical IP based load balancers
> > which help us to get the closest EC2 region and from there on to an
> > availability zone.
>
> > Our application needs to maintain data in memcached. So we have
> > distributed the list of memcached servers
> > across regions. Now we find that this has introduced a signifcant
> > amount of latency.
>
> > From what I have read we may need to replicate the data. But this will
> > introduce attendant problems
> > of consistency which we will have to resolve through a protocol like
> > quorum. I was wondering
> > if memcached has any option of automatically taking care of
> > replication and also ensuring
> > consistency.
>
> > Regards
> > Ganesh
>
> > On Jul 5, 1:33 pm, Dustin <dsalli...@gmail.com> wrote:
> >> On Jul 4, 10:32 pm, TV Ganesh <tvganesh...@gmail.com> wrote:
>
> >> > Dustin,
> >> > I guess you are saying that when we use memcached across
> >> > regions then latencies are a given.
> >> > Is there a way out of this issue. Is it through replication? Does
> >> > memcached have an option for replication?
>
> >> OK, so by "across regions" you're referring to Amazon EC2 regions.
> >> You should note that. :)
>
> >> There's nothing you can do to make light go faster. When you are
> >> communicating between two machines in the same availability zone, you
> >> can expect some, but probably negligible latency for your needs. When
> >> you jump to an entire other region, you should expect an additional
> >> 40ms added to your request processing. It just takes that long for
> >> the message to get there and the response to get back.
>
> >> If you had super magical bidirectional replication that was
> >> asynchronous and you didn't care whatsoever about data consistency,
> >> you could keep an additional copy in both locations and stay roughly
> >> 20ms behind. If you did have this magic, I would strongly recommend
> >> against using it.
>
> >> Knowing what the right answer is requires knowing a lot more about
> >> what your application is doing. For example, why do you need to
> >> pretend like the entire United States of America is one data center?
> >> Is it possible to segment your traffic such that data is localized
> >> into one availability zone, but potentially can fail over to another
> >> in a DR scenario?
