Or, when the client issues a DELETE command, send it to all servers, not
just the one it hashes too?
I agree that memcached servers knowing about each other is a hefty change
and really overkill.
Brandon Ramirez | Office: 585.214.5013 | Fax: 585.295.4848
Software Engineer II | Element K | www.elementk.com
From: Adam Lee <[email protected]>
To: [email protected]
Date: 01/14/2010 05:18 PM
Subject: Re: how to avoid picking up stale data after delete
Sent by: [email protected]
This is an acknowledged problem, but I think that you're over-thinking it.
In general, your memcached servers shouldn't be flapping like that very
often. Second, memcached should be used as a cache, so that in the worst
case, you're going to read slightly outdated data and it shouldn't
completely kill you. Also, rather than your clients deleting cache
entries, you should consider using expiration times on your entries, so
that they're managed by the server.
Redesigning memcached so that servers are aware of each other would be
total overkill and would slow it down greatly. If you absolutely need to
guarantee the integrity of your data, then you probably shouldn't be using
memcached.
Another simple solution would be to change your client code so that once a
server goes down, it's not automatically added back into the pool.
On Thu, Jan 14, 2010 at 4:51 PM, taso <[email protected]> wrote:
Hi,
I have recently started looking at memcached. In my testing I
discovered a scenario in which it's possible to pick up stale data
from
a cache server. By "stale", I mean data that was assumed to be
deleted.
I was wondering if this is a well known problem and whether there's
any
solution or workaround.
Here is the scenario:
Suppose there are 2 memcached instances. Let's call them Server A and
Server B.
1. A and B are both up and running
2. Client sets foo=bar in the cache. Let's suppose it goes to Server
A.
3. Server A goes down
4. Client makes a cache request for "foo". Since A is down, it
attempts
to read from B, which gives us a cache miss.
5. Client sets foo=bar in the cache. This time it goes to B since A
is
down
6. Server A comes back up
7. Client makes a cache request for "foo". It attempts to read from
A,
which is empty since it just came back up, so we get a cache miss
again.
8. Client sets foo=bar in the cache. The data goes to Server A.
(Note
B is still populated from earlier).
9. Client deletes "foo" from the cache. It is deleted from A (but not
B).
10. Server A goes down again.
11. Client reads foo from cache. Since A is down, it reads from
Server
B, and the value is "bar". But "bar" is stale now; it should have
been
deleted, per step 9.
The only way I could envision solving this, is if the memcached
servers
were aware of each other, and could propagate the deletes to their
peers. I suppose this would be a pretty drastic change to the
system.
Am I just missing some simple solution?
Thanks,
Taso
--
awl
