That doesn't solve the problem. Automatic recovery from failover will always result in an inconsistent cache, the above example was with deleting keys, but it could as well have been about modifying the data. What would you do then? Send all set commands to all servers?
The easiest solution is to simply not use automatic recovery from failover. The absolute majority of users probably don't even need failover, but in the end it's a tradeoff between consistency and availability. If 100% availability is more important than 100% consistency, then by all means, use failover and recovery, but then you've made the conscious decision to accept occasional inconsistencies. /Henrik On Fri, Jan 15, 2010 at 13:49, <[email protected]> wrote: > Or, when the client issues a DELETE command, send it to all servers, not > just the one it hashes too? > > I agree that memcached servers knowing about each other is a hefty change > and really overkill. > > > Brandon Ramirez | Office: 585.214.5013 | Fax: 585.295.4848 > Software Engineer II | Element K | www.elementk.com > > > > > > > From: Adam Lee <[email protected]> > > To: [email protected] > > Date: 01/14/2010 05:18 PM > > Subject: Re: how to avoid picking up stale data after delete > > Sent by: [email protected] > > > > > > > This is an acknowledged problem, but I think that you're over-thinking it. > > In general, your memcached servers shouldn't be flapping like that very > often. Second, memcached should be used as a cache, so that in the worst > case, you're going to read slightly outdated data and it shouldn't > completely kill you. Also, rather than your clients deleting cache > entries, you should consider using expiration times on your entries, so > that they're managed by the server. > > Redesigning memcached so that servers are aware of each other would be > total overkill and would slow it down greatly. If you absolutely need to > guarantee the integrity of your data, then you probably shouldn't be using > memcached. > > Another simple solution would be to change your client code so that once a > server goes down, it's not automatically added back into the pool. > > On Thu, Jan 14, 2010 at 4:51 PM, taso <[email protected]> wrote: > Hi, > > I have recently started looking at memcached. In my testing I > discovered a scenario in which it's possible to pick up stale data > from > a cache server. By "stale", I mean data that was assumed to be > deleted. > I was wondering if this is a well known problem and whether there's > any > solution or workaround. > > Here is the scenario: > > Suppose there are 2 memcached instances. Let's call them Server A and > Server B. > 1. A and B are both up and running > 2. Client sets foo=bar in the cache. Let's suppose it goes to Server > A. > 3. Server A goes down > 4. Client makes a cache request for "foo". Since A is down, it > attempts > to read from B, which gives us a cache miss. > 5. Client sets foo=bar in the cache. This time it goes to B since A > is > down > 6. Server A comes back up > 7. Client makes a cache request for "foo". It attempts to read from > A, > which is empty since it just came back up, so we get a cache miss > again. > 8. Client sets foo=bar in the cache. The data goes to Server A. > (Note > B is still populated from earlier). > 9. Client deletes "foo" from the cache. It is deleted from A (but not > B). > 10. Server A goes down again. > 11. Client reads foo from cache. Since A is down, it reads from > Server > B, and the value is "bar". But "bar" is stale now; it should have > been > deleted, per step 9. > > The only way I could envision solving this, is if the memcached > servers > were aware of each other, and could propagate the deletes to their > peers. I suppose this would be a pretty drastic change to the > system. > > Am I just missing some simple solution? > > Thanks, > Taso > > > > -- > awl > > >
