Reviewed: https://review.openstack.org/623011 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3b1463b968426ba8db1c5d458f6461e2375045e8 Submitter: Zuul Branch: master
commit 3b1463b968426ba8db1c5d458f6461e2375045e8 Author: Matt Riedemann <[email protected]> Date: Wed Dec 5 10:46:06 2018 -0500 Use tempest [compute]/build_timeout in evacuate tests Waiting 30 seconds for an evacuate to complete is not enough time on some slower CI test nodes. This change uses the same build timeout configuration from tempest to determine the overall evacuate timeout in our evacuate tests. Change-Id: Ie5935ae54d2cbf1a4272e93815ee5f67d3ffe2eb Closes-Bug: #1806925 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1806925 Title: evacuate test fails due to timeout waiting for evacuate to complete Status in OpenStack Compute (nova): Fix Released Bug description: In the post-test hook in the nova-live-migration job where we test evacuate, we're doing the following: 1. create an image-backed and volume-backed server on the subnode 2. stop libvirtd on the local node 3. run evacuate to see it fail because nova-compute is disabled on the local node 4. restart libvirtd, wait for the local nova-compute service to be enabled, and then evacuate each server In this failure, the evacuate times out because libvirtd is still unavailable on the local node after we started the evacuate: http://logs.openstack.org/54/620154/1/gate/nova-live- migration/f040b76/logs/devstack-gate- post_test_hook.txt.gz#_2018-12-05_10_05_50_130 2018-12-05 10:05:50.130 | + /opt/stack/new/nova/gate/test_evacuate.sh:evacuate_and_wait_for_active:114 : nova evacuate evacuate-test nova-compute on the local host is back up here: Dec 05 10:05:49.341595 ubuntu-xenial-ovh-bhs1-0000944602 nova- compute[16115]: INFO nova.virt.libvirt.driver [None req-e14feea2-2abc- 43cc-b51f-f416f9dd5692 None None] Connection event '1' reason 'None' The evacuate starts here: http://logs.openstack.org/54/620154/1/gate/nova-live- migration/f040b76/logs/screen-n-cpu.txt.gz#_Dec_05_10_05_54_156579 Dec 05 10:05:54.156579 ubuntu-xenial-ovh-bhs1-0000944602 nova- compute[16115]: INFO nova.compute.manager [None req-c2f2a1d3-527f-4885 -8e4f-e82003a6d472 demo admin] [instance: 19ef59e3-de5a-42b2-b0aa- d069702deedf] Evacuating instance After that I don't see any failures, but the evacuation doesn't complete within the 30 second timeout - maybe the timeout isn't long enough? It looks like while we timeout, we're waiting for the network-vif- plugged event from neutron: http://logs.openstack.org/54/620154/1/gate/nova-live- migration/f040b76/logs/screen-n-cpu.txt.gz#_Dec_05_10_06_04_554322 Dec 05 10:06:04.554322 ubuntu-xenial-ovh-bhs1-0000944602 nova- compute[16115]: DEBUG nova.compute.manager [None req- c2f2a1d3-527f-4885-8e4f-e82003a6d472 demo admin] [instance: 19ef59e3 -de5a-42b2-b0aa-d069702deedf] Preparing to wait for external event network-vif-plugged-7d5ba599-9c7a-4e41-9fe4-3aff44a75458 {{(pid=16115) prepare_for_instance_event /opt/stack/new/nova/nova/compute/manager.py:327}} The VIF is plugged here: Dec 05 10:06:04.620986 ubuntu-xenial-ovh-bhs1-0000944602 nova- compute[16115]: INFO os_vif [None req-c2f2a1d3-527f-4885-8e4f- e82003a6d472 demo admin] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:e5:b1:9f,bridge_name='br- int',has_traffic_filtering=True,id=7d5ba599-9c7a- 4e41-9fe4-3aff44a75458,network=Network(22273876-0d80-4450-8913-0102f3f79ccf),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap7d5ba599-9c') And we timeout about a second or so later, but vif plugging usually takes about 5 seconds to get the event back from neutron, and this is a slower ovh node, so our timeout is likely just not long enough. To compare, tempest's compute build_timeout is 300 seconds: https://github.com/openstack/tempest/blob/eac094a8cf834d035316a900107f601adcc42ff5/tempest/config.py#L288 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1806925/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
