Reviewed: https://review.openstack.org/445142 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b8be61eb39dc9f605ad853e5697a8f4bf73b025b Submitter: Jenkins Branch: master
commit b8be61eb39dc9f605ad853e5697a8f4bf73b025b Author: Matthew Booth <[email protected]> Date: Mon Mar 13 18:28:17 2017 +0000 Fix and optimize external_events for multiple cells server_external_events was previously making an api db query and a cell db query for every instance reference. We improve this by making exactly 1 api db query to fetch all instance mappings, and then 1 cell db query per cell to fetch all relevant instances from that cell. Further, it wasn't properly handling the case where events were delivered in one request for multiple instances across cells, which this also fixes. We also document an obtuse edge condition in ComputeAPI.external_instance_event which will cause the current code to break when we support migration between cells. Note this includes a tweak to the SingleCellSimple fixture to mock out the new InstanceMappingList method we use, as well as a fix to the other InstanceMapping mock, which was returning mappings with bogus instance uuids. This patch relies on the results of those being realistic and thus requires those changes. Closes-Bug: #1702959 Co-Authored-By: Dan Smith <[email protected]> Change-Id: If59453f1899e99040c554bcb9ad54c8a506adc56 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1702959 Title: MigrationNotFound in multi-cell setup doing server external events processing Status in OpenStack Compute (nova): Fix Released Bug description: I noticed this in the devstack change testing multi-cell, in the neutron multinode job: http://logs.openstack.org/56/477556/3/check/gate-tempest-dsvm-neutron- multinode-full-ubuntu-xenial- nv/ee3e9b6/logs/screen-n-api.txt.gz?level=TRACE#_Jul_05_20_55_53_749696 We're getting migration not found because we're not targeted to a cell to find the migration: Jul 05 20:55:53.749696 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions [None req-d4ca5b2e-140c-4331-88eb-50bae2de2230 service nova] Unexpected exception in API method: MigrationNotFound: Migration 15 could not be found. Jul 05 20:55:53.749806 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions Traceback (most recent call last): Jul 05 20:55:53.749888 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/openstack/extensions.py", line 336, in wrapped Jul 05 20:55:53.749965 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions return f(*args, **kwargs) Jul 05 20:55:53.750053 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 108, in wrapper Jul 05 20:55:53.750134 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions return func(*args, **kwargs) Jul 05 20:55:53.750222 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/api/openstack/compute/server_external_events.py", line 120, in create Jul 05 20:55:53.750303 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions context, accepted_instances, mappings, accepted_events) Jul 05 20:55:53.750438 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/compute/api.py", line 4046, in external_instance_event Jul 05 20:55:53.750564 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions for host in self._get_relevant_hosts(context, instance): Jul 05 20:55:53.750647 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/compute/api.py", line 4072, in _get_relevant_hosts Jul 05 20:55:53.750722 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions migration = objects.Migration.get_by_id(context, migration_id) Jul 05 20:55:53.750798 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper Jul 05 20:55:53.750873 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) Jul 05 20:55:53.750949 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/objects/migration.py", line 106, in get_by_id Jul 05 20:55:53.751023 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions db_migration = db.migration_get(context, migration_id) Jul 05 20:55:53.751107 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/db/api.py", line 535, in migration_get Jul 05 20:55:53.751187 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions return IMPL.migration_get(context, migration_id) Jul 05 20:55:53.751264 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 265, in wrapped Jul 05 20:55:53.751338 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions return f(context, *args, **kwargs) Jul 05 20:55:53.751414 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions File "/opt/stack/new/nova/nova/db/sqlalchemy/api.py", line 4719, in migration_get Jul 05 20:55:53.751492 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions raise exception.MigrationNotFound(migration_id=id) Jul 05 20:55:53.751568 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions MigrationNotFound: Migration 15 could not be found. Jul 05 20:55:53.751640 ubuntu-xenial-2-node-osic-cloud1-s3500-9668742 [email protected][27811]: ERROR nova.api.openstack.extensions https://github.com/openstack/nova/blob/56cd608d3a199dcb02ac2ae071ff3057241259da/nova/compute/api.py#L4301 We need to use the mappings dict passed to external_instance_event to mutate the context while looking up the migration. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1702959/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
