This is an acknowledged problem, but I think that you're over-thinking it. In general, your memcached servers shouldn't be flapping like that very often. Second, memcached should be used as a cache, so that in the worst case, you're going to read slightly outdated data and it shouldn't completely kill you. Also, rather than your clients deleting cache entries, you should consider using expiration times on your entries, so that they're managed by the server.
Redesigning memcached so that servers are aware of each other would be total overkill and would slow it down greatly. If you absolutely need to guarantee the integrity of your data, then you probably shouldn't be using memcached. Another simple solution would be to change your client code so that once a server goes down, it's not automatically added back into the pool. On Thu, Jan 14, 2010 at 4:51 PM, taso <[email protected]> wrote: > Hi, > > I have recently started looking at memcached. In my testing I > discovered a scenario in which it's possible to pick up stale data > from > a cache server. By "stale", I mean data that was assumed to be > deleted. > I was wondering if this is a well known problem and whether there's > any > solution or workaround. > > Here is the scenario: > > Suppose there are 2 memcached instances. Let's call them Server A and > Server B. > 1. A and B are both up and running > 2. Client sets foo=bar in the cache. Let's suppose it goes to Server > A. > 3. Server A goes down > 4. Client makes a cache request for "foo". Since A is down, it > attempts > to read from B, which gives us a cache miss. > 5. Client sets foo=bar in the cache. This time it goes to B since A > is > down > 6. Server A comes back up > 7. Client makes a cache request for "foo". It attempts to read from > A, > which is empty since it just came back up, so we get a cache miss > again. > 8. Client sets foo=bar in the cache. The data goes to Server A. > (Note > B is still populated from earlier). > 9. Client deletes "foo" from the cache. It is deleted from A (but not > B). > 10. Server A goes down again. > 11. Client reads foo from cache. Since A is down, it reads from > Server > B, and the value is "bar". But "bar" is stale now; it should have > been > deleted, per step 9. > > The only way I could envision solving this, is if the memcached > servers > were aware of each other, and could propagate the deletes to their > peers. I suppose this would be a pretty drastic change to the > system. > > Am I just missing some simple solution? > > Thanks, > Taso > -- awl
