Public bug reported: Description =========== The unit-test "nova.tests.unit.virt.test_virt_drivers.LibvirtConnTestCase.test_set_admin_password" fails. I could narrow it down to this commit: https://github.com/openstack/nova/commit/735f710
move os_vif.initialize() to nova-compute start os_vif.initialize() was previously executed during module load. This means it was entirely possible that it was run before things like logging were actually set up in the expected way. Move this back into execution time instead of load time to ensure that logging is actually setup. Changes need to be made to tests which make assumptions about os_vif objects to manually initialize os_vif when it will be used. os_vif objects can't be created until it is initialized, so some delayed object creation is also done in test_vif.py. Closes-Bug: #1615676 Change-Id: I89fe5c5b3d762f3a3238b587685df85d15ee56c4 Steps to reproduce ================== A chronological list of steps which will bring off the issue you noticed: * Clone/pull nova * tox -e py27 nova.tests.unit.virt.test_virt_drivers.LibvirtConnTestCase.test_set_admin_password -r Expected result =============== Test should succeed Actual result ============= ============================== Failed 1 tests - output below: ============================== nova.tests.unit.virt.test_virt_drivers.LibvirtConnTestCase.test_set_admin_password ---------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "nova/tests/unit/virt/test_virt_drivers.py", line 58, in wrapped_func return f(self, *args, **kwargs) File "/home/markus/git/nova/.tox/py27/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched return func(*args, **keywargs) File "nova/tests/unit/virt/test_virt_drivers.py", line 980, in test_set_admin_password instance, network_info = self._get_running_instance(obj=True) File "nova/tests/unit/virt/test_virt_drivers.py", line 261, in _get_running_instance [], 'herp', network_info=network_info) File "nova/virt/libvirt/driver.py", line 2602, in spawn write_to_disk=True) File "nova/virt/libvirt/driver.py", line 4651, in _get_guest_xml context) File "nova/virt/libvirt/driver.py", line 4483, in _get_guest_config flavor, virt_type, self._host) File "nova/virt/libvirt/vif.py", line 507, in get_config vif_obj = os_vif_util.nova_to_osvif_vif(vif) File "nova/network/os_vif_util.py", line 369, in nova_to_osvif_vif vifobj = func(vif) File "nova/network/os_vif_util.py", line 248, in _nova_to_osvif_vif_bridge objects.vif.VIFBridge, AttributeError: 'module' object has no attribute 'vif' Captured pythonlogging: ~~~~~~~~~~~~~~~~~~~~~~~ 2016-10-13 10:12:16,176 INFO [248_add_expire_reservations_index] Skipped adding reservations_deleted_expire_idx because an equivalent index already exists. 2016-10-13 10:12:18,153 INFO [os_brick.initiator.connectors.disco] Init DISCO connector 2016-10-13 10:12:18,153 WARNING [os_brick.initiator.connectors.remotefs] Connection details not present. RemoteFsClient may not initialize properly. 2016-10-13 10:12:18,209 INFO [nova.virt.osinfo] Cannot load Libosinfo: (No module named gi.repository.Libosinfo) 2016-10-13 10:12:18,209 INFO [nova.virt.libvirt.driver] Creating image 2016-10-13 10:12:18,258 INFO [nova.virt.libvirt.driver] Connection event '1' reason 'None' 2016-10-13 10:12:18,260 INFO [nova.virt.libvirt.host] Libvirt host capabilities <capabilities> <host> <uuid>cef19ce0-0ca2-11df-855d-b19fbce37686</uuid> <cpu> <arch>x86_64</arch> <model>Penryn</model> <vendor>Intel</vendor> <topology sockets='1' cores='2' threads='1'/> <feature name='xtpr'/> <feature name='tm2'/> <feature name='est'/> <feature name='vmx'/> <feature name='ds_cpl'/> <feature name='monitor'/> <feature name='pbe'/> <feature name='tm'/> <feature name='ht'/> <feature name='ss'/> <feature name='acpi'/> <feature name='ds'/> <feature name='vme'/> </cpu> <migration_features> <live/> <uri_transports> <uri_transport>tcp</uri_transport> </uri_transports> </migration_features> <secmodel> <model>apparmor</model> <doi>0</doi> </secmodel> </host> <guest> <os_type>hvm</os_type> <arch name='i686'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu</emulator> <machine>pc-0.14</machine> <machine canonical='pc-0.14'>pc</machine> <machine>pc-0.13</machine> <machine>pc-0.12</machine> <machine>pc-0.11</machine> <machine>pc-0.10</machine> <machine>isapc</machine> <domain type='qemu'> </domain> <domain type='kvm'> <emulator>/usr/bin/kvm</emulator> <machine>pc-0.14</machine> <machine canonical='pc-0.14'>pc</machine> <machine>pc-0.13</machine> <machine>pc-0.12</machine> <machine>pc-0.11</machine> <machine>pc-0.10</machine> <machine>isapc</machine> </domain> </arch> <features> <cpuselection/> <deviceboot/> <pae/> <nonpae/> <acpi default='on' toggle='yes'/> <apic default='on' toggle='no'/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='x86_64'> <wordsize>64</wordsize> <emulator>/usr/bin/qemu-system-x86_64</emulator> <machine>pc-0.14</machine> <machine canonical='pc-0.14'>pc</machine> <machine>pc-0.13</machine> <machine>pc-0.12</machine> <machine>pc-0.11</machine> <machine>pc-0.10</machine> <machine>isapc</machine> <domain type='qemu'> </domain> <domain type='kvm'> <emulator>/usr/bin/kvm</emulator> <machine>pc-0.14</machine> <machine canonical='pc-0.14'>pc</machine> <machine>pc-0.13</machine> <machine>pc-0.12</machine> <machine>pc-0.11</machine> <machine>pc-0.10</machine> <machine>isapc</machine> </domain> </arch> <features> <cpuselection/> <deviceboot/> <acpi default='on' toggle='yes'/> <apic default='on' toggle='no'/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='armv7l'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu-system-arm</emulator> <machine>integratorcp</machine> <machine>vexpress-a9</machine> <machine>syborg</machine> <machine>musicpal</machine> <machine>mainstone</machine> <machine>n800</machine> <machine>n810</machine> <machine>n900</machine> <machine>cheetah</machine> <machine>sx1</machine> <machine>sx1-v1</machine> <machine>beagle</machine> <machine>beaglexm</machine> <machine>tosa</machine> <machine>akita</machine> <machine>spitz</machine> <machine>borzoi</machine> <machine>terrier</machine> <machine>connex</machine> <machine>verdex</machine> <machine>lm3s811evb</machine> <machine>lm3s6965evb</machine> <machine>realview-eb</machine> <machine>realview-eb-mpcore</machine> <machine>realview-pb-a8</machine> <machine>realview-pbx-a9</machine> <machine>versatilepb</machine> <machine>versatileab</machine> <domain type='qemu'> </domain> </arch> <features> <deviceboot/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='mips'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu-system-mips</emulator> <machine>malta</machine> <machine>mipssim</machine> <machine>magnum</machine> <machine>pica61</machine> <machine>mips</machine> <domain type='qemu'> </domain> </arch> <features> <deviceboot/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='mipsel'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu-system-mipsel</emulator> <machine>malta</machine> <machine>mipssim</machine> <machine>magnum</machine> <machine>pica61</machine> <machine>mips</machine> <domain type='qemu'> </domain> </arch> <features> <deviceboot/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='sparc'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu-system-sparc</emulator> <machine>SS-5</machine> <machine>leon3_generic</machine> <machine>SS-10</machine> <machine>SS-600MP</machine> <machine>SS-20</machine> <machine>Voyager</machine> <machine>LX</machine> <machine>SS-4</machine> <machine>SPARCClassic</machine> <machine>SPARCbook</machine> <machine>SS-1000</machine> <machine>SS-2000</machine> <machine>SS-2</machine> <domain type='qemu'> </domain> </arch> </guest> <guest> <os_type>hvm</os_type> <arch name='ppc'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu-system-ppc</emulator> <machine>g3beige</machine> <machine>virtex-ml507</machine> <machine>mpc8544ds</machine> <machine canonical='bamboo-0.13'>bamboo</machine> <machine>bamboo-0.13</machine> <machine>bamboo-0.12</machine> <machine>ref405ep</machine> <machine>taihu</machine> <machine>mac99</machine> <machine>prep</machine> <domain type='qemu'> </domain> </arch> <features> <deviceboot/> </features> </guest> </capabilities> ====== Totals ====== Ran: 1 tests in 30.0000 sec. - Passed: 0 - Skipped: 0 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 1 Sum of execute time for each test: 3.6373 sec. ============== Worker Balance ============== - Worker 0 (1 tests) => 0:00:03.637283 No tests were successful during the run ERROR: InvocationError: '/usr/bin/bash tools/pretty_tox.sh nova.tests.unit.virt.test_virt_drivers.LibvirtConnTestCase.test_set_admin_password' ______________________________________________________ summary ______________________________________________________ ERROR: py27: commands failed Environment =========== 1. Nova master (Ocata dev cycle) [10:13:39 markus@oc5730007623 ~/git/nova ] $ git log -1 commit bc1b11fdc2c140be916ee3b3b31993cbd6d13ac6 Merge: 9be53df 0fafb81 Author: Jenkins <jenk...@review.openstack.org> Date: Thu Oct 13 03:04:45 2016 +0000 Merge "libvirt: cleanup never used migratable flag checking" Logs & Configs ============== N/A ** Affects: nova Importance: Undecided Assignee: Markus Zoeller (markus_z) (mzoeller) Status: In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1632987 Title: _nova_to_osvif_vif_bridge: 'module' object has no attribute 'vif' Status in OpenStack Compute (nova): In Progress Bug description: Description =========== The unit-test "nova.tests.unit.virt.test_virt_drivers.LibvirtConnTestCase.test_set_admin_password" fails. I could narrow it down to this commit: https://github.com/openstack/nova/commit/735f710 move os_vif.initialize() to nova-compute start os_vif.initialize() was previously executed during module load. This means it was entirely possible that it was run before things like logging were actually set up in the expected way. Move this back into execution time instead of load time to ensure that logging is actually setup. Changes need to be made to tests which make assumptions about os_vif objects to manually initialize os_vif when it will be used. os_vif objects can't be created until it is initialized, so some delayed object creation is also done in test_vif.py. Closes-Bug: #1615676 Change-Id: I89fe5c5b3d762f3a3238b587685df85d15ee56c4 Steps to reproduce ================== A chronological list of steps which will bring off the issue you noticed: * Clone/pull nova * tox -e py27 nova.tests.unit.virt.test_virt_drivers.LibvirtConnTestCase.test_set_admin_password -r Expected result =============== Test should succeed Actual result ============= ============================== Failed 1 tests - output below: ============================== nova.tests.unit.virt.test_virt_drivers.LibvirtConnTestCase.test_set_admin_password ---------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "nova/tests/unit/virt/test_virt_drivers.py", line 58, in wrapped_func return f(self, *args, **kwargs) File "/home/markus/git/nova/.tox/py27/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched return func(*args, **keywargs) File "nova/tests/unit/virt/test_virt_drivers.py", line 980, in test_set_admin_password instance, network_info = self._get_running_instance(obj=True) File "nova/tests/unit/virt/test_virt_drivers.py", line 261, in _get_running_instance [], 'herp', network_info=network_info) File "nova/virt/libvirt/driver.py", line 2602, in spawn write_to_disk=True) File "nova/virt/libvirt/driver.py", line 4651, in _get_guest_xml context) File "nova/virt/libvirt/driver.py", line 4483, in _get_guest_config flavor, virt_type, self._host) File "nova/virt/libvirt/vif.py", line 507, in get_config vif_obj = os_vif_util.nova_to_osvif_vif(vif) File "nova/network/os_vif_util.py", line 369, in nova_to_osvif_vif vifobj = func(vif) File "nova/network/os_vif_util.py", line 248, in _nova_to_osvif_vif_bridge objects.vif.VIFBridge, AttributeError: 'module' object has no attribute 'vif' Captured pythonlogging: ~~~~~~~~~~~~~~~~~~~~~~~ 2016-10-13 10:12:16,176 INFO [248_add_expire_reservations_index] Skipped adding reservations_deleted_expire_idx because an equivalent index already exists. 2016-10-13 10:12:18,153 INFO [os_brick.initiator.connectors.disco] Init DISCO connector 2016-10-13 10:12:18,153 WARNING [os_brick.initiator.connectors.remotefs] Connection details not present. RemoteFsClient may not initialize properly. 2016-10-13 10:12:18,209 INFO [nova.virt.osinfo] Cannot load Libosinfo: (No module named gi.repository.Libosinfo) 2016-10-13 10:12:18,209 INFO [nova.virt.libvirt.driver] Creating image 2016-10-13 10:12:18,258 INFO [nova.virt.libvirt.driver] Connection event '1' reason 'None' 2016-10-13 10:12:18,260 INFO [nova.virt.libvirt.host] Libvirt host capabilities <capabilities> <host> <uuid>cef19ce0-0ca2-11df-855d-b19fbce37686</uuid> <cpu> <arch>x86_64</arch> <model>Penryn</model> <vendor>Intel</vendor> <topology sockets='1' cores='2' threads='1'/> <feature name='xtpr'/> <feature name='tm2'/> <feature name='est'/> <feature name='vmx'/> <feature name='ds_cpl'/> <feature name='monitor'/> <feature name='pbe'/> <feature name='tm'/> <feature name='ht'/> <feature name='ss'/> <feature name='acpi'/> <feature name='ds'/> <feature name='vme'/> </cpu> <migration_features> <live/> <uri_transports> <uri_transport>tcp</uri_transport> </uri_transports> </migration_features> <secmodel> <model>apparmor</model> <doi>0</doi> </secmodel> </host> <guest> <os_type>hvm</os_type> <arch name='i686'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu</emulator> <machine>pc-0.14</machine> <machine canonical='pc-0.14'>pc</machine> <machine>pc-0.13</machine> <machine>pc-0.12</machine> <machine>pc-0.11</machine> <machine>pc-0.10</machine> <machine>isapc</machine> <domain type='qemu'> </domain> <domain type='kvm'> <emulator>/usr/bin/kvm</emulator> <machine>pc-0.14</machine> <machine canonical='pc-0.14'>pc</machine> <machine>pc-0.13</machine> <machine>pc-0.12</machine> <machine>pc-0.11</machine> <machine>pc-0.10</machine> <machine>isapc</machine> </domain> </arch> <features> <cpuselection/> <deviceboot/> <pae/> <nonpae/> <acpi default='on' toggle='yes'/> <apic default='on' toggle='no'/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='x86_64'> <wordsize>64</wordsize> <emulator>/usr/bin/qemu-system-x86_64</emulator> <machine>pc-0.14</machine> <machine canonical='pc-0.14'>pc</machine> <machine>pc-0.13</machine> <machine>pc-0.12</machine> <machine>pc-0.11</machine> <machine>pc-0.10</machine> <machine>isapc</machine> <domain type='qemu'> </domain> <domain type='kvm'> <emulator>/usr/bin/kvm</emulator> <machine>pc-0.14</machine> <machine canonical='pc-0.14'>pc</machine> <machine>pc-0.13</machine> <machine>pc-0.12</machine> <machine>pc-0.11</machine> <machine>pc-0.10</machine> <machine>isapc</machine> </domain> </arch> <features> <cpuselection/> <deviceboot/> <acpi default='on' toggle='yes'/> <apic default='on' toggle='no'/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='armv7l'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu-system-arm</emulator> <machine>integratorcp</machine> <machine>vexpress-a9</machine> <machine>syborg</machine> <machine>musicpal</machine> <machine>mainstone</machine> <machine>n800</machine> <machine>n810</machine> <machine>n900</machine> <machine>cheetah</machine> <machine>sx1</machine> <machine>sx1-v1</machine> <machine>beagle</machine> <machine>beaglexm</machine> <machine>tosa</machine> <machine>akita</machine> <machine>spitz</machine> <machine>borzoi</machine> <machine>terrier</machine> <machine>connex</machine> <machine>verdex</machine> <machine>lm3s811evb</machine> <machine>lm3s6965evb</machine> <machine>realview-eb</machine> <machine>realview-eb-mpcore</machine> <machine>realview-pb-a8</machine> <machine>realview-pbx-a9</machine> <machine>versatilepb</machine> <machine>versatileab</machine> <domain type='qemu'> </domain> </arch> <features> <deviceboot/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='mips'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu-system-mips</emulator> <machine>malta</machine> <machine>mipssim</machine> <machine>magnum</machine> <machine>pica61</machine> <machine>mips</machine> <domain type='qemu'> </domain> </arch> <features> <deviceboot/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='mipsel'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu-system-mipsel</emulator> <machine>malta</machine> <machine>mipssim</machine> <machine>magnum</machine> <machine>pica61</machine> <machine>mips</machine> <domain type='qemu'> </domain> </arch> <features> <deviceboot/> </features> </guest> <guest> <os_type>hvm</os_type> <arch name='sparc'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu-system-sparc</emulator> <machine>SS-5</machine> <machine>leon3_generic</machine> <machine>SS-10</machine> <machine>SS-600MP</machine> <machine>SS-20</machine> <machine>Voyager</machine> <machine>LX</machine> <machine>SS-4</machine> <machine>SPARCClassic</machine> <machine>SPARCbook</machine> <machine>SS-1000</machine> <machine>SS-2000</machine> <machine>SS-2</machine> <domain type='qemu'> </domain> </arch> </guest> <guest> <os_type>hvm</os_type> <arch name='ppc'> <wordsize>32</wordsize> <emulator>/usr/bin/qemu-system-ppc</emulator> <machine>g3beige</machine> <machine>virtex-ml507</machine> <machine>mpc8544ds</machine> <machine canonical='bamboo-0.13'>bamboo</machine> <machine>bamboo-0.13</machine> <machine>bamboo-0.12</machine> <machine>ref405ep</machine> <machine>taihu</machine> <machine>mac99</machine> <machine>prep</machine> <domain type='qemu'> </domain> </arch> <features> <deviceboot/> </features> </guest> </capabilities> ====== Totals ====== Ran: 1 tests in 30.0000 sec. - Passed: 0 - Skipped: 0 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 1 Sum of execute time for each test: 3.6373 sec. ============== Worker Balance ============== - Worker 0 (1 tests) => 0:00:03.637283 No tests were successful during the run ERROR: InvocationError: '/usr/bin/bash tools/pretty_tox.sh nova.tests.unit.virt.test_virt_drivers.LibvirtConnTestCase.test_set_admin_password' ______________________________________________________ summary ______________________________________________________ ERROR: py27: commands failed Environment =========== 1. Nova master (Ocata dev cycle) [10:13:39 markus@oc5730007623 ~/git/nova ] $ git log -1 commit bc1b11fdc2c140be916ee3b3b31993cbd6d13ac6 Merge: 9be53df 0fafb81 Author: Jenkins <jenk...@review.openstack.org> Date: Thu Oct 13 03:04:45 2016 +0000 Merge "libvirt: cleanup never used migratable flag checking" Logs & Configs ============== N/A To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1632987/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp