Reviewed: https://review.opendev.org/582408 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=16e163053ca39886f11fdb8a3af10a28619fc105 Submitter: Zuul Branch: master
commit 16e163053ca39886f11fdb8a3af10a28619fc105 Author: melanie witt <[email protected]> Date: Thu Jul 12 21:48:23 2018 +0000 Don't generate service UUID for deleted services In Pike, we added a UUID field to services and during an upgrade from Ocata => Pike, when instances are accessed joined with their associated services, we generate a UUID for the services on-the-fly. This causes a problem in the scenario where an operator upgrades their cluster and has old, deleted services with hostnames matching existing services associated with instances. When we go to generate the service UUID for the old, deleted service, we hit a ServiceTooOld exception. This addresses the problem by not bothering to generate a UUID for a deleted service. One alternative would be to exclude deleted services when we join the 'instances' and 'services' tables, but I'm not sure whether that approach might cause unintended effects where service information that used to be viewable for instances becomes hidden. Closes-Bug: #1778305 Closes-Bug: #1764556 Change-Id: I347096a527c257075cefe7b81210622f6cd87daf ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1764556 Title: "nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: Confirmed Status in OpenStack Compute (nova) queens series: Confirmed Status in OpenStack Compute (nova) rocky series: Confirmed Status in OpenStack Compute (nova) stein series: Confirmed Bug description: We had a testcase where we booted an instance on Newton, migrated it off the compute node, deleted the compute node (and service), upgraded to Pike, created a new compute node with the same name, and migrated the instance back to the compute node. At this point the "nova list" command failed with exception.ServiceNotFound. It appears that since the Service has no UUID the _from_db_object() routine will try to add it, but the service.save() call fails because the service in question has been deleted. I reproduced the issue with stable/pike devstack. I booted an instance, then created a fake entry in the "services" table without a UUID so the table looked like this: mysql> select * from services; +---------------------+---------------------+---------------------+----+----------+----------------+-----------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+ | created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason | last_seen_up | forced_down | version | uuid | +---------------------+---------------------+---------------------+----+----------+----------------+-----------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+ | 2018-02-20 16:10:07 | 2018-04-16 22:10:46 | NULL | 1 | devstack | nova-conductor | conductor | 477364 | 0 | 0 | NULL | 2018-04-16 22:10:46 | 0 | 22 | c041d7cf-5047-4014-b50c-3ba6b5d95097 | | 2018-02-20 16:10:10 | 2018-04-16 22:10:54 | NULL | 2 | devstack | nova-compute | compute | 477149 | 0 | 0 | NULL | 2018-04-16 22:10:54 | 0 | 22 | d0cfb63c-8b59-4b65-bb7e-6b89acd3fe35 | | 2018-02-20 16:10:10 | 2018-04-16 20:29:33 | 2018-04-16 20:30:33 | 3 | devstack | nova-compute | compute | 476432 | 0 | 3 | NULL | 2018-04-16 20:30:33 | 0 | 22 | NULL | +---------------------+---------------------+---------------------+----+----------+----------------+-----------+--------------+----------+---------+-----------------+---------------------+-------------+---------+--------------------------------------+ At this point, running "nova show <uuid>" worked fine, but running "nova list" failed: stack@devstack:~/devstack$ nova list ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'nova.exception.ServiceNotFound'> (HTTP 500) (Request-ID: req-b7e1b5f9-e7b4-4ccf-ba28-e8b3e1acd2f6) The nova-api log looked like this: Apr 16 22:11:00 devstack [email protected][4258]: DEBUG nova.compute.api [None req-b7e1b5f9-e7b4-4ccf-ba28-e8b3e1acd2f6 demo demo] Listing 1000 instances in cell 09eb515f-9906-40bf-9be6-63b5e6ee279a(cell1) {{(pid=4261) _get_instances_by_filters_all_cells /opt/stack/nova/nova/compute/api.py:2559}} Apr 16 22:11:00 devstack [email protected][4258]: DEBUG oslo_concurrency.lockutils [None req-b7e1b5f9-e7b4-4ccf-ba28-e8b3e1acd2f6 demo demo] Lock "09eb515f-9906-40bf-9be6-63b5e6ee279a" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s {{(pid=4261) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:270}} Apr 16 22:11:00 devstack [email protected][4258]: DEBUG oslo_concurrency.lockutils [None req-b7e1b5f9-e7b4-4ccf-ba28-e8b3e1acd2f6 demo demo] Lock "09eb515f-9906-40bf-9be6-63b5e6ee279a" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.000s {{(pid=4261) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282}} Apr 16 22:11:00 devstack [email protected][4258]: DEBUG nova.objects.service [None req-b7e1b5f9-e7b4-4ccf-ba28-e8b3e1acd2f6 demo demo] Generated UUID 4368a7ff-f589-4197-b0b9-d2afdb71ca33 for service 3 {{(pid=4261) _from_db_object /opt/stack/nova/nova/objects/service.py:245}} Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions [None req-b7e1b5f9-e7b4-4ccf-ba28-e8b3e1acd2f6 demo demo] Unexpected exception in API method: ServiceNotFound: Service 3 could not be found. Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions Traceback (most recent call last): Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 336, in wrapped Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions return f(*args, **kwargs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions return func(*args, **kwargs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 181, in wrapper Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions return func(*args, **kwargs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 168, in detail Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions servers = self._get_servers(req, is_detail=True) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 311, in _get_servers Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions sort_keys=sort_keys, sort_dirs=sort_dirs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2468, in get_all Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions sort_dirs=sort_dirs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2565, in _get_instances_by_filters_all_cells Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions **kwargs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2596, in _get_instances_by_filters Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions expected_attrs=fields, sort_keys=sort_keys, sort_dirs=sort_dirs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 1252, in get_by_filters Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions expected_attrs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 1199, in _make_instance_list Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions expected_attrs=expected_attrs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 448, in _from_db_object Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions objects.Service, db_inst['services']) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 1121, in obj_make_list Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions **extra_args) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/service.py", line 246, in _from_db_object Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions service.save() Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/service.py", line 363, in save Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions db_service = db.service_update(self._context, self.id, updates) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/api.py", line 189, in service_update Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions return IMPL.service_update(context, service_id, values) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions ectxt.value = e.inner_exc Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions self.force_reraise() Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions return f(*args, **kwargs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 250, in wrapped Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions return f(context, *args, **kwargs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 610, in service_update Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions service_ref = service_get(context, service_id) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 265, in wrapped Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions return f(context, *args, **kwargs) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 472, in service_get Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions raise exception.ServiceNotFound(service_id=service_id) Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions ServiceNotFound: Service 3 could not be found. Apr 16 22:11:00 devstack [email protected][4258]: ERROR nova.api.openstack.extensions Apr 16 22:11:00 devstack [email protected][4258]: INFO nova.api.openstack.wsgi [None req-b7e1b5f9-e7b4-4ccf-ba28-e8b3e1acd2f6 demo demo] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. Apr 16 22:11:00 devstack [email protected][4258]: <class 'nova.exception.ServiceNotFound'> Apr 16 22:11:00 devstack [email protected][4258]: DEBUG nova.api.openstack.wsgi [None req-b7e1b5f9-e7b4-4ccf-ba28-e8b3e1acd2f6 demo demo] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. Apr 16 22:11:00 devstack [email protected][4258]: <class 'nova.exception.ServiceNotFound'> {{(pid=4261) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1029}} Apr 16 22:11:00 devstack [email protected][4258]: INFO nova.api.openstack.requestlog [None req-b7e1b5f9-e7b4-4ccf-ba28-e8b3e1acd2f6 demo demo] 128.224.186.226 "GET /compute/v2.1/servers/detail" status: 500 len: 204 microversion: 2.53 time: 0.131473 Apr 16 22:11:00 devstack [email protected][4258]: [pid: 4261|app: 0|req: 6/12] 128.224.186.226 () {64 vars in 1290 bytes} [Mon Apr 16 22:11:00 2018] GET /compute/v2.1/servers/detail => generated 204 bytes in 132 msecs (HTTP/1.1 500) 9 headers in 393 bytes (1 switches on core 0) To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1764556/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
