** Changed in: keystone
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1401108
Title:
Memcache pool locks on release
Status in OpenStack Identity (Keystone):
Fix Released
Bug description:
At some point Keystone stops processing HTTP requests and HAProxy
moves the server out of pool. Once Keystones on all controllers stuck
HAProxy starts to respond with HTTP 503 error.
Symptoms:
~~~~~~~~~
HAProxy log:
2014-12-08T11:01:06.135849+00:00 info: 192.168.0.23:34984
[08/Dec/2014:11:01:06.135] keystone-2 keystone-2/<NOSRV> 0/-1/-1/-1/0 503 212 -
- SC-- 366/0/0/0/0 0/0 "POST /v2.0/tokens HTTP/1.1"
Keystone keeps accepting TCP connection, but hangs at HTTP level
(meaning that telnet works, but curl hangs)
root@node-21:/var/log/keystone# strace -p 26330
Process 26330 attached - interrupt to quit
futex(0xe50420, FUTEX_WAIT_PRIVATE, 0, NULL
The last logged message from stuck process:
2014-12-05 17:01:20.749 26330 DEBUG keystone.common.cache._memcache_pool [-]
Memcached pool 39391504, thread 139781728069376: Acquiring connection
_debug_logger
/usr/lib/python2.7/dist-packages/keystone/common/cache/_memcache_pool.py:91
Memcache pool has asymmetric behaviour on aquire and release of a connection:
If pool is depleted a new connection is created
If pool has reached it's maximum size that connection excess still being
pushed back to the pool causing thread lock.
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1401108/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
