Matt, are you able to recreate this with master or mitaka? ** Also affects: keystone/liberty Importance: Undecided Status: New
** Summary changed: - memcache pool reap issue (stable/liberty) + memcache pool reap issue -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1570158 Title: memcache pool reap issue Status in OpenStack Identity (keystone): New Status in OpenStack Identity (keystone) liberty series: New Bug description: There seems to be a code error in the memcache pool cleanup. I'm seeing this on stable/liberty (built as of 2 weeks ago). I don't have a specific reproducer for this, it just seems to happen. Looking at the code I don't really understand how this can happen. 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi [req-30e23a7d-754d-4a89-904f-1804a20029f2 - - - - -] deque index out of range 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi Traceback (most recent call last): 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/keystone/common/wsgi.py", line 248, in __call__ 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi result = method(context, **params) 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/keystone/common/controller.py", line 138, in inner 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi context['subject_token_id'])) 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/keystone/token/provider.py", line 188, in validate_token 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi token = self._validate_token(unique_id) 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/dogpile/cache/region.py", line 1053, in decorate 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi should_cache_fn) 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/dogpile/cache/region.py", line 657, in get_or_create 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi async_creator) as value: 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 158, in __enter__ 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi return self._enter() 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 91, in _enter 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi value = value_fn() 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/dogpile/cache/region.py", line 610, in get_value 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi value = self.backend.get(key) 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 161, in get 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi value = self.client.get(key) 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/keystone/common/cache/backends/memcache_pool.py", line 36, in _run_method 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi return getattr(client, __name)(*args, **kwargs) 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi self.gen.next() 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/keystone/common/cache/_memcache_pool.py", line 132, in acquire 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi self._drop_expired_connections() 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi File "/venv/local/lib/python2.7/site-packages/keystone/common/cache/_memcache_pool.py", line 166, in _drop_expired_connections 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi while self.queue and self.queue[0].ttl < now: 2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi IndexError: deque index out of range To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1570158/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp