Public bug reported: We are running openstack mitaka out of clean installation (no upgrade). All was functioning correctly until recently nova started to perform updates like this
2016-11-23 10:48:09.065 17498 INFO nova.service [req-c91ec40f-2dc2 - - - - -] Updating service version for nova-conductor on nova-tlc01.specs.de from 15 to 9 The full error log from /var/log/nova/nova-conductor.log is looks like 2016-11-23 10:48:08.967 17499 INFO nova.service [-] Starting conductor node (version 13.1.2) 2016-11-23 10:48:09.065 17498 INFO nova.service [req-c91ec40f-2dc2- - - - - -] Updating service version for nova-conductor on nova-tlc01.specs.de from 15 to 9 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service [req-c91ec40f-2dc2-4ab0-ad61-1ab1f9d7bbc9 - - - - -] Error starting thread. 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service Traceback (most recent call last): 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 680, in run_service 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service service.start() 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 195, in start 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service _update_service_ref(self.service_ref) 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 151, in _update_service_ref 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service service.save() 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 223, in wrapper 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service return fn(self, *args, **kwargs) 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 291, in save 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service self._check_minimum_version() 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 275, in _check_minimum_version 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service minver=minver) 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service ServiceTooOld: This service is older (v9) than the minimum (v15) version of the rest of the deployment. Unable to continue. 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service I checked the changes in recent versions of nova/service.py https://fossies.org/diffs/nova/13.1.1_vs_13.1.2/nova/service.py-diff.html Not sure, but looks like the problem is in this function under nova/service.py def _update_service_ref(service): if service.version != service_obj.SERVICE_VERSION: LOG.info(_LI('Updating service version for %(binary)s on ' '%(host)s from %(old)i to %(new)i'), {'binary': service.binary, 'host': service.host, 'old': service.version, 'new': service_obj.SERVICE_VERSION}) service.version = service_obj.SERVICE_VERSION service.save() ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1644186 Title: nova service performs wrong update from version 15 to 9 Status in OpenStack Compute (nova): New Bug description: We are running openstack mitaka out of clean installation (no upgrade). All was functioning correctly until recently nova started to perform updates like this 2016-11-23 10:48:09.065 17498 INFO nova.service [req-c91ec40f-2dc2 - - - - -] Updating service version for nova-conductor on nova- tlc01.specs.de from 15 to 9 The full error log from /var/log/nova/nova-conductor.log is looks like 2016-11-23 10:48:08.967 17499 INFO nova.service [-] Starting conductor node (version 13.1.2) 2016-11-23 10:48:09.065 17498 INFO nova.service [req-c91ec40f-2dc2- - - - - -] Updating service version for nova-conductor on nova-tlc01.specs.de from 15 to 9 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service [req-c91ec40f-2dc2-4ab0-ad61-1ab1f9d7bbc9 - - - - -] Error starting thread. 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service Traceback (most recent call last): 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 680, in run_service 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service service.start() 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 195, in start 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service _update_service_ref(self.service_ref) 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/service.py", line 151, in _update_service_ref 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service service.save() 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 223, in wrapper 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service return fn(self, *args, **kwargs) 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 291, in save 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service self._check_minimum_version() 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 275, in _check_minimum_version 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service minver=minver) 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service ServiceTooOld: This service is older (v9) than the minimum (v15) version of the rest of the deployment. Unable to continue. 2016-11-23 10:48:09.068 17498 ERROR oslo_service.service I checked the changes in recent versions of nova/service.py https://fossies.org/diffs/nova/13.1.1_vs_13.1.2/nova/service.py-diff.html Not sure, but looks like the problem is in this function under nova/service.py def _update_service_ref(service): if service.version != service_obj.SERVICE_VERSION: LOG.info(_LI('Updating service version for %(binary)s on ' '%(host)s from %(old)i to %(new)i'), {'binary': service.binary, 'host': service.host, 'old': service.version, 'new': service_obj.SERVICE_VERSION}) service.version = service_obj.SERVICE_VERSION service.save() To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1644186/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
