Reviewed: https://review.opendev.org/582408 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=16e163053ca39886f11fdb8a3af10a28619fc105 Submitter: Zuul Branch: master
commit 16e163053ca39886f11fdb8a3af10a28619fc105 Author: melanie witt <[email protected]> Date: Thu Jul 12 21:48:23 2018 +0000 Don't generate service UUID for deleted services In Pike, we added a UUID field to services and during an upgrade from Ocata => Pike, when instances are accessed joined with their associated services, we generate a UUID for the services on-the-fly. This causes a problem in the scenario where an operator upgrades their cluster and has old, deleted services with hostnames matching existing services associated with instances. When we go to generate the service UUID for the old, deleted service, we hit a ServiceTooOld exception. This addresses the problem by not bothering to generate a UUID for a deleted service. One alternative would be to exclude deleted services when we join the 'instances' and 'services' tables, but I'm not sure whether that approach might cause unintended effects where service information that used to be viewable for instances becomes hidden. Closes-Bug: #1778305 Closes-Bug: #1764556 Change-Id: I347096a527c257075cefe7b81210622f6cd87daf ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1778305 Title: Nova may erronously look up service version of a deleted service, when hostname have been reused Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: Confirmed Status in OpenStack Compute (nova) queens series: Confirmed Status in OpenStack Compute (nova) rocky series: Confirmed Status in OpenStack Compute (nova) stein series: Confirmed Bug description: Prerequisites: - A compute node running an old version of nova has been deleted. (In our case, version 9) - The hostname of said compute node has been reused, and has been upgraded as per normal. (To version 16) - The services table in the nova database contains both the old and the new node defined, where the deleted one are clearly marked as deleted - and with the old version specified in the version column. The new node also exist, upgraded as it is. - One has at least one instance running on the upgraded node. - Perform upgrade from ocata to pike - Any projects with instances running on the upgraded node, may erronously get an error message that "ERROR (BadRequest): This service is older (v9) than the minimum (v16) version of the rest of the deployment. Unable to continue. (HTTP 400) (Request-ID: req-3e0ababe-e09b-4ef8-ba3a-43060bc1f807)" --- when performing 'nova list'. Example of how this may look in the database: MariaDB [nova]> SELECT * FROM services WHERE host = 'node11.acme.org'; +---------------------+---------------------+---------------------+-----+-----------------+--------------+---------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+ | created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason | last_seen_up | forced_down | version | uuid | +---------------------+---------------------+---------------------+-----+-----------------+--------------+---------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+ | 2017-10-17 13:06:10 | 2018-06-22 21:42:42 | NULL | 179 | node11.acme.org | nova-compute | compute | 2138069 | 0 | 0 | NULL | 2018-06-22 21:42:42 | 0 | 22 | 63e1cb55-ee00-4cb8-b304-160dd5c45fdd | | 2016-08-13 08:20:05 | 2016-11-15 00:01:21 | 2016-11-27 15:11:30 | 104 | node11.acme.org | nova-compute | compute | 796220 | 1 | 104 | NULL | 2016-11-15 00:01:21 | 0 | 9 | NULL | +---------------------+---------------------+---------------------+-----+-----------------+--------------+---------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+ 2 rows in set (0.01 sec) Removing the old service from the database is an effective workaround for this problem. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1778305/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
