Public bug reported: When running with the Ubuntu Xenial Mitaka packages, I'm seeing the following behaviour:
Following the release notes at http://docs.openstack.org/releasenotes/nova/mitaka.html I remove the old option "memcached_servers" and set up a cache section with [cache] enabled = true memcache_servers = host1:11211,host2:11211,host3:11211 The result is that there are errors logged when creating a token: 2016-04-19 10:17:33.501 15952 WARNING nova.consoleauth.manager [req-ad043de9-616d-4cbb-a93f-fc5e345f1f53 1b0106d4f339406bb7012a83162ba5f2 e3c253d3e8344a8796e70bc4f96b6166 - - -] Token: dd0e7bbc-c593-421f-a56b-52578aaec20e failed to save into memcached. 2016-04-19 10:17:33.503 15952 WARNING nova.consoleauth.manager [req-ad043de9-616d-4cbb-a93f-fc5e345f1f53 1b0106d4f339406bb7012a83162ba5f2 e3c253d3e8344a8796e70bc4f96b6166 - - -] Instance: d3267abb-258d-4b5d-a22e-8e3b0a39905c failed to save into memcached Only after a lot of debugging it turns out that the default backend for oslo_cache is dogpile.cache.null, implying that no values get cached and token validation always fails. Only after adding [cache] backend = oslo_cache.memcache_pool does console authentication start working. Strangely though, the above warning messages are still being logged in the working setup, which made debugging this even more difficult. So I suggest the following fixes: 1. Change the text of the warnings from "failed to save into memcached" to "failed to save into cache", as with the change to using oslo_cache, there may be other backends in use instead of memcached. 2. Either override the default of using the null backend or refuse to run with it or at the very least give a big fat warning that the configuration can not work. 3. Stop generating the warning messages when the data got in fact saved into cache properly. Package versions for reference: # dpkg -l | grep nova ii nova-api-metadata 2:13.0.0-0ubuntu2 all OpenStack Compute - metadata API frontend ii nova-api-os-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - OpenStack Compute API frontend ii nova-cert 2:13.0.0-0ubuntu2 all OpenStack Compute - certificate management ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files ii nova-conductor 2:13.0.0-0ubuntu2 all OpenStack Compute - conductor service ii nova-consoleauth 2:13.0.0-0ubuntu2 all OpenStack Compute - Console Authenticator ii nova-scheduler 2:13.0.0-0ubuntu2 all OpenStack Compute - virtual machine scheduler ii nova-spiceproxy 2:13.0.0-0ubuntu2 all OpenStack Compute - spice html5 proxy ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7 # dpkg -l | grep oslo ii python-oslo.cache 1.6.0-2 all cache storage for Openstack projects - Python 2.7 ii python-oslo.concurrency 3.7.0-2 all concurrency and locks for OpenStack projects - Python 2.x ii python-oslo.config 1:3.9.0-3 all Common code for Openstack Projects (configuration API) - Python 2.x ii python-oslo.context 2.2.0-2 all WSGI context helpers for OpenStack - Python 2.x ii python-oslo.db 4.7.0-2ubuntu1 all database connectivity to the different backends and helper utils - Python 2.x ii python-oslo.i18n 3.5.0-2 all Oslo Internationalization Utilities - Python 2.x ii python-oslo.log 3.2.0-2 all OpenStack logging configuration library - Python 2.x ii python-oslo.messaging 4.6.1-2ubuntu1 all oslo messaging library - Python 2.x ii python-oslo.middleware 3.8.0-2 all various WSGI middleware components for OpenStack - Python 2.x ii python-oslo.policy 1.6.0-2 all RBAC policy enforcement library for OpenStack - Python 2.x ii python-oslo.reports 1.7.0-2 all reports serialized in various data types - Python 2.7 ii python-oslo.rootwrap 4.1.0-2 all allows fine filtering of shell commands to run as root - Python 2.x ii python-oslo.serialization 2.4.0-2 all utilities for serialization , especially JSON - Python 2.x ii python-oslo.service 1.8.0-1ubuntu1 all library for running OpenStack services - Python 2.x ii python-oslo.utils 3.8.0-2 all set of utility functions for OpenStack - Python 2.x ii python-oslo.versionedobjects 1.8.0-1 all deals with DB schema versions and code expectations - Python 2.x ii python-oslo.vmware 2.5.0-2 all VMware library for OpenStack projects - Python 2.7 ** Affects: nova Importance: Undecided Assignee: Dr. Jens Rosenboom (j-rosenboom-j) Status: In Progress ** Changed in: nova Assignee: (unassigned) => Dr. Jens Rosenboom (j-rosenboom-j) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1572062 Title: nova-consoleauth doesn't play well with memcached Status in OpenStack Compute (nova): In Progress Bug description: When running with the Ubuntu Xenial Mitaka packages, I'm seeing the following behaviour: Following the release notes at http://docs.openstack.org/releasenotes/nova/mitaka.html I remove the old option "memcached_servers" and set up a cache section with [cache] enabled = true memcache_servers = host1:11211,host2:11211,host3:11211 The result is that there are errors logged when creating a token: 2016-04-19 10:17:33.501 15952 WARNING nova.consoleauth.manager [req-ad043de9-616d-4cbb-a93f-fc5e345f1f53 1b0106d4f339406bb7012a83162ba5f2 e3c253d3e8344a8796e70bc4f96b6166 - - -] Token: dd0e7bbc-c593-421f-a56b-52578aaec20e failed to save into memcached. 2016-04-19 10:17:33.503 15952 WARNING nova.consoleauth.manager [req-ad043de9-616d-4cbb-a93f-fc5e345f1f53 1b0106d4f339406bb7012a83162ba5f2 e3c253d3e8344a8796e70bc4f96b6166 - - -] Instance: d3267abb-258d-4b5d-a22e-8e3b0a39905c failed to save into memcached Only after a lot of debugging it turns out that the default backend for oslo_cache is dogpile.cache.null, implying that no values get cached and token validation always fails. Only after adding [cache] backend = oslo_cache.memcache_pool does console authentication start working. Strangely though, the above warning messages are still being logged in the working setup, which made debugging this even more difficult. So I suggest the following fixes: 1. Change the text of the warnings from "failed to save into memcached" to "failed to save into cache", as with the change to using oslo_cache, there may be other backends in use instead of memcached. 2. Either override the default of using the null backend or refuse to run with it or at the very least give a big fat warning that the configuration can not work. 3. Stop generating the warning messages when the data got in fact saved into cache properly. Package versions for reference: # dpkg -l | grep nova ii nova-api-metadata 2:13.0.0-0ubuntu2 all OpenStack Compute - metadata API frontend ii nova-api-os-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - OpenStack Compute API frontend ii nova-cert 2:13.0.0-0ubuntu2 all OpenStack Compute - certificate management ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files ii nova-conductor 2:13.0.0-0ubuntu2 all OpenStack Compute - conductor service ii nova-consoleauth 2:13.0.0-0ubuntu2 all OpenStack Compute - Console Authenticator ii nova-scheduler 2:13.0.0-0ubuntu2 all OpenStack Compute - virtual machine scheduler ii nova-spiceproxy 2:13.0.0-0ubuntu2 all OpenStack Compute - spice html5 proxy ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7 # dpkg -l | grep oslo ii python-oslo.cache 1.6.0-2 all cache storage for Openstack projects - Python 2.7 ii python-oslo.concurrency 3.7.0-2 all concurrency and locks for OpenStack projects - Python 2.x ii python-oslo.config 1:3.9.0-3 all Common code for Openstack Projects (configuration API) - Python 2.x ii python-oslo.context 2.2.0-2 all WSGI context helpers for OpenStack - Python 2.x ii python-oslo.db 4.7.0-2ubuntu1 all database connectivity to the different backends and helper utils - Python 2.x ii python-oslo.i18n 3.5.0-2 all Oslo Internationalization Utilities - Python 2.x ii python-oslo.log 3.2.0-2 all OpenStack logging configuration library - Python 2.x ii python-oslo.messaging 4.6.1-2ubuntu1 all oslo messaging library - Python 2.x ii python-oslo.middleware 3.8.0-2 all various WSGI middleware components for OpenStack - Python 2.x ii python-oslo.policy 1.6.0-2 all RBAC policy enforcement library for OpenStack - Python 2.x ii python-oslo.reports 1.7.0-2 all reports serialized in various data types - Python 2.7 ii python-oslo.rootwrap 4.1.0-2 all allows fine filtering of shell commands to run as root - Python 2.x ii python-oslo.serialization 2.4.0-2 all utilities for serialization , especially JSON - Python 2.x ii python-oslo.service 1.8.0-1ubuntu1 all library for running OpenStack services - Python 2.x ii python-oslo.utils 3.8.0-2 all set of utility functions for OpenStack - Python 2.x ii python-oslo.versionedobjects 1.8.0-1 all deals with DB schema versions and code expectations - Python 2.x ii python-oslo.vmware 2.5.0-2 all VMware library for OpenStack projects - Python 2.7 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1572062/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
