Actually looks like the vif is plugged successfully. This is where it starts:
http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm-functional- convg-mysql-lbaasv2-py35-ubuntu- xenial/17c2da9/logs/screen-n-cpu.txt.gz?level=DEBUG#_May_28_04_09_48_125012 And this is where it says it was successful: http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm-functional- convg-mysql-lbaasv2-py35-ubuntu- xenial/17c2da9/logs/screen-n-cpu.txt.gz?level=DEBUG#_May_28_04_09_48_221125 And then it times out 5 minutes later: http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm-functional- convg-mysql-lbaasv2-py35-ubuntu- xenial/17c2da9/logs/screen-n-cpu.txt.gz?level=DEBUG#_May_28_04_14_48_559598 Looks like the port is plugged on the neutron side here: http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm-functional- convg-mysql-lbaasv2-py35-ubuntu- xenial/17c2da9/logs/screen-q-svc.txt.gz#_May_28_04_09_48_412466 But I don't see a network-vif-plugged event. I do see the network-vif- unplugged event 5 minutes later: http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm-functional- convg-mysql-lbaasv2-py35-ubuntu- xenial/17c2da9/logs/screen-q-svc.txt.gz#_May_28_04_14_50_528198 There was a network-vif-plugged event before though: http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm-functional- convg-mysql-lbaasv2-py35-ubuntu- xenial/17c2da9/logs/screen-q-svc.txt.gz#_May_28_04_09_32_689843 I wonder if ^ is from the original spawn and then the 2nd vif plug is from the rebuild action, and that's why we don't get a network-vif- plugged event from neutron, because it thinks it's already plugged to that instance on that host? ** Also affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1694371 Title: test_stack_snapshot_restore failing intermittently Status in heat: New Status in neutron: New Status in OpenStack Compute (nova): New Bug description: It seems like the server is going to ERROR state during rebuild. traceback: 2017-05-28 04:34:37.767757 | 2017-05-28 04:34:37.767 | Captured traceback: 2017-05-28 04:34:37.768828 | 2017-05-28 04:34:37.768 | ~~~~~~~~~~~~~~~~~~~ 2017-05-28 04:34:37.770099 | 2017-05-28 04:34:37.769 | b'Traceback (most recent call last):' 2017-05-28 04:34:37.771108 | 2017-05-28 04:34:37.770 | b' File "/opt/stack/new/heat/heat_integrationtests/functional/test_snapshot_restore.py", line 74, in test_stack_snapshot_restore' 2017-05-28 04:34:37.772364 | 2017-05-28 04:34:37.772 | b' self.stack_restore(stack_identifier, snapshot_id)' 2017-05-28 04:34:37.773407 | 2017-05-28 04:34:37.773 | b' File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 626, in stack_restore' 2017-05-28 04:34:37.774541 | 2017-05-28 04:34:37.774 | b' self._wait_for_stack_status(stack_id, wait_for_status)' 2017-05-28 04:34:37.775568 | 2017-05-28 04:34:37.775 | b' File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 357, in _wait_for_stack_status' 2017-05-28 04:34:37.776642 | 2017-05-28 04:34:37.776 | b' fail_regexp):' 2017-05-28 04:34:37.778354 | 2017-05-28 04:34:37.778 | b' File "/opt/stack/new/heat/heat_integrationtests/common/test.py", line 321, in _verify_status' 2017-05-28 04:34:37.779448 | 2017-05-28 04:34:37.779 | b' stack_status_reason=stack.stack_status_reason)' 2017-05-28 04:34:37.780644 | 2017-05-28 04:34:37.780 | b"heat_integrationtests.common.exceptions.StackBuildErrorException: Stack StackSnapshotRestoreTest-1374582671/7fb8f800-1545-4e34-a6fa-3e2adbf4443a is in RESTORE_FAILED status due to 'Error: resources.my_server: Rebuilding server failed, status 'ERROR''" 2017-05-28 04:34:37.782119 | 2017-05-28 04:34:37.781 | b'' Noticed at: http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm- functional-convg-mysql-lbaasv2-py35-ubuntu- xenial/17c2da9/console.html#_2017-05-28_04_34_37_753094 Looks like a nova issue from the below traceback. http://logs.openstack.org/16/462216/16/check/gate-heat-dsvm- functional-convg-mysql-lbaasv2-py35-ubuntu- xenial/17c2da9/logs/screen-n-cpu.txt.gz?level=ERROR#_May_28_04_14_49_044455 May 28 04:14:49.042877 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/compute/manager.py", line 6758, in _error_out_instance_on_exception May 28 04:14:49.042955 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] yield May 28 04:14:49.043027 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/compute/manager.py", line 2814, in rebuild_instance May 28 04:14:49.043100 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] bdms, recreate, on_shared_storage, preserve_ephemeral) May 28 04:14:49.043197 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/compute/manager.py", line 2855, in _do_rebuild_instance_with_claim May 28 04:14:49.043299 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] self._do_rebuild_instance(*args, **kwargs) May 28 04:14:49.043384 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/compute/manager.py", line 2977, in _do_rebuild_instance May 28 04:14:49.043458 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] self._rebuild_default_impl(**kwargs) May 28 04:14:49.043534 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/compute/manager.py", line 2714, in _rebuild_default_impl May 28 04:14:49.043611 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] block_device_info=new_block_device_info) May 28 04:14:49.044132 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2741, in spawn May 28 04:14:49.044223 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] destroy_disks_on_failure=True) May 28 04:14:49.044299 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5152, in _create_domain_and_network May 28 04:14:49.044374 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] raise exception.VirtualInterfaceCreateException() May 28 04:14:49.044455 ubuntu-xenial-osic-cloud1-s3700-9024798 nova-compute[26709]: ERROR nova.compute.manager [instance: 45105d34-b970-4ced-968c-a1c4ead5b282] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed To manage notifications about this bug go to: https://bugs.launchpad.net/heat/+bug/1694371/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
