Yes, if you use automatic recovery from failover your cache can get unsynchronized as different parts of your application discover that a previous failing server is now back up at different points in time.
If synchronization is very important to your application, make sure you don't use automatic recovery from failover, and you won't get this problem. The flipside to that is that when you want to put back servers in the cluster, you need to restart your application so that all parts of it get the updated server list at the same time. Another way of solving the problem is to not use failover at all. If your application is fine with more cache misses as long as one of your cache servers is down, then that solution is the best. You will never have synchronization problems, and you don't have to restart your application to bring back servers into the cluster. In reality though, I have never seen a memcached server crash in production. It is very, very stable. Normally you don't have to worry about what happens if one server goes down, because they never do. /Henrik On Wed, Feb 24, 2010 at 01:13, mcevikce <[email protected]> wrote: > Thank you for the prompt reply it is a very helpful article. But it > also raises some questions. > Lets use the example from that article. > > Cache A contains values 1,4 > Cache B contains values 2 > Cache C contains values 3 > > Questions 1-) If cache A does down, would cache B will now have 4,1and > 2? If so this is not equally distributed. cache B contains 3 values > (4,1,2) and Cache C has only one value (3). Lets assume if cache A > actually contained thousands of values, shifting these values to only > cache B will truly unbalance the distribution. > > Questions-2) Lets use the same example as above for starters. > Cache A contains values 1,4 > Cache B contains values 2 > Cache C contains values 3 > Lets say because of a network issue Cache A was not accessible but the > server is still up. Client will probably move values to cache B. Now > cache B has values 4,1,2. Lets say the content of values 1,4 changed. > Cache B has the new content for values. NOW the network problem is > fixed and Cache A again available. Will client get the values 1 and 4 > from Cache A or Cache B? If it gets it from cache A, these values will > be stale because new content for these values are now in Cache B. > > On Feb 23, 5:49 pm, Matt Ingenthron <[email protected]> wrote: > > mcevikce wrote: > > > I understand that memcached servers do not communicate with each > > > other. I also understand that consistent hashing algorithm in client > > > libraries endure that data is distributed evenly between the servers. > > > What I am not clear about is: > > > Scenario 1: > > > Three server nodes running. key1 is stores on server #1. Server #1 > > > goes down. > > > 1-) Client makes a request on key1 stored on failed server. My > > > assumption is that cache will get a miss and value for key1 will be > > > retrieved from DB and stored in server #2. > > > 2-) Now dead server is brought up. Client makes the request on a > > > key1 again. Does client get the key1 from server #2 then store it back > > > into server #1? How does re balancing occur? > > > > > Scenario 2: > > > Three nodes running. Another node is added. How does memcached > > > redistribute values to new node? I know that servers themselves do not > > > communicate with each other. > > > > The data isn't so much redistributed as it is re-populated from the > > primary source to the new location in the consistent hash. The old > > items in memory in the old positions will eventually expire or LRU out. > > > > The key here is that the clients all have a consistent view and shared > > hashing algorithm pointing to the same servers. If you add a new node, > > for some portion of keys there will initally be misses, but then things > > will balance out later. > > > > Have a look at: > http://www.last.fm/user/RJ/journal/2007/04/10/rz_libketama_-_a_consis...http://weblogs.java.net/blog/2007/11/27/consistent-hashing > > > > Hope that helps, > > > > - Matt >
