Reviewed: https://review.openstack.org/634334 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=03e4e3ce1357123f9f53a3901a17a64f160653a4 Submitter: Zuul Branch: master
commit 03e4e3ce1357123f9f53a3901a17a64f160653a4 Author: Matt Riedemann <[email protected]> Date: Thu Jan 31 17:36:47 2019 -0500 Fix race in test_volume_swap_server_with_error Change I03c8e8225e51fd80580772752c0b292987e34218 added another notification to the list checked in test_volume_swap_server_with_error and then change Id587967ea4f9980c292492e2f659bf55fb037b28 relied on checking the last notification in the list (compute.exception) but per the note in the test, that last exception can race to show up by the time we check the size of the notification list which can result in an IndexError. This change fixes the race by just waiting for the compute.exception notification rather than the racy _wait_until_swap_volume_error which depended on the cinder fixture swap_error variable (which is removed in this change). Change-Id: I8165fcd98e11b9155640559a36ae90a38f63c0dd Closes-Bug: #1814177 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1814177 Title: nova.tests.functional.notification_sample_tests.test_instance.TestInstanceNotificationSample.test_volume_swap_server_with_error intermittently fails with IndexError Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) rocky series: Confirmed Bug description: Seen here: http://logs.openstack.org/86/633086/1/gate/nova-tox-functional/8a2c597 /job-output.txt.gz#_2019-01-31_20_37_18_577225 2019-01-31 20:37:18.641058 | ubuntu-xenial | 2019-01-31 20:37:17,911 ERROR [nova.compute.manager] Failed to connect to volume 9c6d9c2d-7a8f-4c80-938d-3bf062b8d489 with volume at /dev/sdb 2019-01-31 20:37:18.641191 | ubuntu-xenial | Traceback (most recent call last): 2019-01-31 20:37:18.641370 | ubuntu-xenial | File "nova/compute/manager.py", line 5710, in _swap_volume 2019-01-31 20:37:18.641508 | ubuntu-xenial | bdm, new_attachment_id, mountpoint) 2019-01-31 20:37:18.641707 | ubuntu-xenial | File "nova/compute/manager.py", line 5697, in _init_volume_connection 2019-01-31 20:37:18.641850 | ubuntu-xenial | new_cinfo['serial'] = new_volume_id 2019-01-31 20:37:18.642025 | ubuntu-xenial | TypeError: 'tuple' object does not support item assignment 2019-01-31 20:37:18.642275 | ubuntu-xenial | 2019-01-31 20:37:18,369 ERROR [oslo_messaging.rpc.server] Exception during message handling 2019-01-31 20:37:18.642402 | ubuntu-xenial | Traceback (most recent call last): 2019-01-31 20:37:18.642783 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-01-31 20:37:18.642931 | ubuntu-xenial | res = self.dispatcher.dispatch(message) 2019-01-31 20:37:18.643307 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2019-01-31 20:37:18.643490 | ubuntu-xenial | return self._do_dispatch(endpoint, method, ctxt, args) 2019-01-31 20:37:18.643874 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2019-01-31 20:37:18.644007 | ubuntu-xenial | result = func(ctxt, **new_args) 2019-01-31 20:37:18.644175 | ubuntu-xenial | File "nova/exception_wrapper.py", line 79, in wrapped 2019-01-31 20:37:18.644314 | ubuntu-xenial | function_name, call_dict, binary, tb) 2019-01-31 20:37:18.644739 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-01-31 20:37:18.644850 | ubuntu-xenial | self.force_reraise() 2019-01-31 20:37:18.645209 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-01-31 20:37:18.645365 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb) 2019-01-31 20:37:18.645532 | ubuntu-xenial | File "nova/exception_wrapper.py", line 69, in wrapped 2019-01-31 20:37:18.645668 | ubuntu-xenial | return f(self, context, *args, **kw) 2019-01-31 20:37:18.645864 | ubuntu-xenial | File "nova/compute/utils.py", line 1141, in decorated_function 2019-01-31 20:37:18.646051 | ubuntu-xenial | return function(self, context, *args, **kwargs) 2019-01-31 20:37:18.646263 | ubuntu-xenial | File "nova/compute/manager.py", line 216, in decorated_function 2019-01-31 20:37:18.646406 | ubuntu-xenial | kwargs['instance'], e, sys.exc_info()) 2019-01-31 20:37:18.646755 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-01-31 20:37:18.646864 | ubuntu-xenial | self.force_reraise() 2019-01-31 20:37:18.647224 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-01-31 20:37:18.647379 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb) 2019-01-31 20:37:18.647564 | ubuntu-xenial | File "nova/compute/manager.py", line 204, in decorated_function 2019-01-31 20:37:18.647721 | ubuntu-xenial | return function(self, context, *args, **kwargs) 2019-01-31 20:37:18.647894 | ubuntu-xenial | File "nova/compute/manager.py", line 5868, in swap_volume 2019-01-31 20:37:18.647997 | ubuntu-xenial | is_cinder_migration) 2019-01-31 20:37:18.648170 | ubuntu-xenial | File "nova/compute/manager.py", line 5766, in _swap_volume 2019-01-31 20:37:18.648288 | ubuntu-xenial | context, new_attachment_id) 2019-01-31 20:37:18.648676 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-01-31 20:37:18.648784 | ubuntu-xenial | self.force_reraise() 2019-01-31 20:37:18.649144 | ubuntu-xenial | File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-01-31 20:37:18.649298 | ubuntu-xenial | six.reraise(self.type_, self.value, self.tb) 2019-01-31 20:37:18.649473 | ubuntu-xenial | File "nova/compute/manager.py", line 5710, in _swap_volume 2019-01-31 20:37:18.649606 | ubuntu-xenial | bdm, new_attachment_id, mountpoint) 2019-01-31 20:37:18.649809 | ubuntu-xenial | File "nova/compute/manager.py", line 5697, in _init_volume_connection 2019-01-31 20:37:18.649966 | ubuntu-xenial | new_cinfo['serial'] = new_volume_id 2019-01-31 20:37:18.650149 | ubuntu-xenial | TypeError: 'tuple' object does not support item assignment 2019-01-31 20:37:18.650208 | ubuntu-xenial | 2019-01-31 20:37:18.650254 | ubuntu-xenial | 2019-01-31 20:37:18.650337 | ubuntu-xenial | Captured traceback: 2019-01-31 20:37:18.650423 | ubuntu-xenial | ~~~~~~~~~~~~~~~~~~~ 2019-01-31 20:37:18.650545 | ubuntu-xenial | Traceback (most recent call last): 2019-01-31 20:37:18.650846 | ubuntu-xenial | File "nova/tests/functional/notification_sample_tests/test_instance.py", line 1443, in test_volume_swap_server_with_error 2019-01-31 20:37:18.651028 | ubuntu-xenial | tb2 = fake_notifier.VERSIONED_NOTIFICATIONS[7]['payload'][ 2019-01-31 20:37:18.651153 | ubuntu-xenial | IndexError: list index out of range The TypeError is expected, it's part of the swap.volume.error notification test. The bug is the IndexError in the expected notifications. We should probably debug that with a patch to check the length and dump the notifications if the length isn't what we expect. http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22tb2%20%3D%20fake_notifier.VERSIONED_NOTIFICATIONS%5B7%5D%5B'payload'%5D%5B%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=10d 6 hits in 10 days, check and gate, all failures. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1814177/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
