Public bug reported: We saw it quite often recently that grenade jobs are failing due to missing memcache module while keystone is starting.
Error message: 2019-03-29 00:22:39.361 | + lib/keystone:bootstrap_keystone:564 : /usr/local/bin/keystone-manage bootstrap --bootstrap-username admin --bootstrap-password secretadmin --bootstrap-project-name admin --bootstrap-role-name admin --bootstrap-service-name keystone --bootstrap-region-id RegionOne --bootstrap-admin-url http://10.209.34.128/identity --bootstrap-public-url http://10.209.34.128/identity 2019-03-29 00:22:41.201 | [01;31mCRITICAL keystone [[00;36m-[01;31m] [01;35m[01;31mUnhandled error[00m: ModuleNotFoundError: No module named 'memcache' 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00mTraceback (most recent call last): 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/usr/local/bin/keystone-manage", line 10, in <module> 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m sys.exit(main()) 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/cmd/manage.py", line 41, in main 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m cli.main(argv=sys.argv, developer_config_file=developer_config) 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 1216, in main 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m CONF.command.cmd_class.main() 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 177, in main 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m klass = cls() 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 66, in __init__ 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m self.bootstrapper = bootstrap.Bootstrapper() 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/cmd/bootstrap.py", line 31, in __init__ 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m backends.load_backends() 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/server/backends.py", line 41, in load_backends 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m cache.configure_cache() 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/common/cache/core.py", line 124, in configure_cache 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m cache.configure_cache_region(CONF, region) 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/oslo_cache/core.py", line 235, in configure_cache_region 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m '%s.' % conf.cache.config_prefix) 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 591, in configure_from_config 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m "%sreplace_existing_backend" % prefix, False), 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 423, in configure 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m _config_prefix 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/api.py", line 89, in from_config_dict 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m for key in config_dict 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 208, in __init__ 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m super(MemcacheArgs, self).__init__(arguments) 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 108, in __init__ 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m self._imports() 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 287, in _imports 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m import memcache # noqa 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00mModuleNotFoundError: No module named 'memcache' Logstash query which I used to find similar examples: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ModuleNotFoundError%3A%20No%20module%20named%20'memcache'%5C%22 Looks that it happens at least for 10 runs in last 7 days. ** Affects: keystone Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1822251 Title: grenade job fail due to missing memcache module Status in OpenStack Identity (keystone): New Bug description: We saw it quite often recently that grenade jobs are failing due to missing memcache module while keystone is starting. Error message: 2019-03-29 00:22:39.361 | + lib/keystone:bootstrap_keystone:564 : /usr/local/bin/keystone-manage bootstrap --bootstrap-username admin --bootstrap-password secretadmin --bootstrap-project-name admin --bootstrap-role-name admin --bootstrap-service-name keystone --bootstrap-region-id RegionOne --bootstrap-admin-url http://10.209.34.128/identity --bootstrap-public-url http://10.209.34.128/identity 2019-03-29 00:22:41.201 | [01;31mCRITICAL keystone [[00;36m-[01;31m] [01;35m[01;31mUnhandled error[00m: ModuleNotFoundError: No module named 'memcache' 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00mTraceback (most recent call last): 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/usr/local/bin/keystone-manage", line 10, in <module> 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m sys.exit(main()) 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/cmd/manage.py", line 41, in main 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m cli.main(argv=sys.argv, developer_config_file=developer_config) 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 1216, in main 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m CONF.command.cmd_class.main() 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 177, in main 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m klass = cls() 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/cmd/cli.py", line 66, in __init__ 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m self.bootstrapper = bootstrap.Bootstrapper() 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/cmd/bootstrap.py", line 31, in __init__ 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m backends.load_backends() 2019-03-29 00:22:41.201 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/server/backends.py", line 41, in load_backends 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m cache.configure_cache() 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/opt/stack/old/keystone/keystone/common/cache/core.py", line 124, in configure_cache 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m cache.configure_cache_region(CONF, region) 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/oslo_cache/core.py", line 235, in configure_cache_region 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m '%s.' % conf.cache.config_prefix) 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 591, in configure_from_config 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m "%sreplace_existing_backend" % prefix, False), 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 423, in configure 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m _config_prefix 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/api.py", line 89, in from_config_dict 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m for key in config_dict 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 208, in __init__ 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m super(MemcacheArgs, self).__init__(arguments) 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 108, in __init__ 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m self._imports() 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/backends/memcached.py", line 287, in _imports 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00m import memcache # noqa 2019-03-29 00:22:41.202 | ERROR keystone [01;35m[00mModuleNotFoundError: No module named 'memcache' Logstash query which I used to find similar examples: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ModuleNotFoundError%3A%20No%20module%20named%20'memcache'%5C%22 Looks that it happens at least for 10 runs in last 7 days. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1822251/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
