Hi,
Scenario: I have 2 memcached nodes and python client (pylibmc from github
master, libmemcached 1.0.3 from tarball) configured as follows:
behaviors = {'remove_failed': 2, 'retry_timeout': 10, 'ketama': True,
'no_block': True, 'tcp_nodelay': False, 'cas': True, 'connect_timeout':
1000, 'dead_timeout': 10}
1. I create client with binary connection and create 20 keys. Some of those
keys are stored on one node and some on another.
2. I start infinite loop that tries to get() values for all those keys one
by one.
3. When I shutdown one node, I'm getting error 47 (server has failed and is
disabled until timed retry). Despite having remove_failed and retry_timeout
set to low values, I'm getting error 47 for few minutes on each get request.
4. After few minutes I'm getting error 47 and 35 (server is marked dead)
every 10 seconds (retry_timeout).
5. Now I start memcached that was down and everything is back to normal
(after dead_timeout).
Now let's modify above scenario. Instead of invoking get() in our infinite
loop I just set() new random keys. After turning off one node, I only get 1
error 5 (server has failed and is disabled until timed retry) and then
error 5 (server is marked dead). I don't know why those error codes are
different than in get() scenario. Anyway, while my loop continues, I'm
getting server dead error every 10 seconds. Then I start memcached again,
but I'm still getting error 5 (server dead). Dead_timeout behavior doesn't
seem to have any effect when not using get() operations.
Any ideas? Anyone seen that before?
Best regards,
Pawel
