Public bug reported: I have a devstack change up which runs tempest with nova + the lxc backend and it fails some instances on boot because the nbd device is in use or the rootfs mount is busy:
https://review.openstack.org/#/c/219448/ http://logs.openstack.org/48/219448/1/check/gate-tempest-dsvm- full/bc0fc2a/logs/screen-n-cpu.txt.gz?level=TRACE It looks like we're leaking nbd devices and failing to tear down instances properly, e.g.: 2015-09-01 22:10:50.141 ERROR nova.virt.disk.api [req-7e5384a8-34f3-4e3a-98cd-b4bb544804a7 tempest-ServersAdminTestJSON-276960936 tempest-ServersAdminTestJSON-718419646] Failed to mount container filesystem '<nova.virt.disk.api._DiskImage object at 0x7f94ec58cd90>' on '/opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs': -- Failed to mount filesystem: Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd4 /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs Exit code: 32 Stdout: u'' Stderr: u'mount: /dev/nbd4 already mounted or /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs busy\n' 2015-09-01 22:10:50.142 ERROR nova.compute.manager [req-7e5384a8-34f3-4e3a-98cd-b4bb544804a7 tempest-ServersAdminTestJSON-276960936 tempest-ServersAdminTestJSON-718419646] [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Instance failed to spawn 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Traceback (most recent call last): 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/compute/manager.py", line 2138, in _build_resources 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] yield resources 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/compute/manager.py", line 2008, in _build_and_run_instance 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] block_device_info=block_device_info) 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2449, in spawn 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] block_device_info=block_device_info) 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4521, in _create_domain_and_network 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] block_device_info, disk_info): 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] return self.gen.next() 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4430, in _lxc_disk_handler 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] block_device_info, disk_info) 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4382, in _create_domain_setup_lxc 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] container_dir=container_dir) 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/virt/disk/api.py", line 452, in setup_container 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] raise exception.NovaException(img.errors) 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] NovaException: 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] -- 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Failed to mount filesystem: Unexpected error while running command. 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd4 /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Exit code: 32 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Stdout: u'' 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Stderr: u'mount: /dev/nbd4 already mounted or /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs busy\n' 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] 2015-09-01 22:10:50.025 DEBUG nova.virt.disk.mount.api [req-7e5384a8-34f3-4e3a-98cd-b4bb544804a7 tempest-ServersAdminTestJSON-276960936 tempest-ServersAdminTestJSON-718419646] Failed to mount filesystem: Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd4 /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs Exit code: 32 Stdout: u'' Stderr: u'mount: /dev/nbd4 already mounted or /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs busy\n' mnt_dev /opt/stack/new/nova/nova/virt/disk/mount/api.py:237 etc ** Affects: nova Importance: Undecided Status: New ** Tags: libvirt lxc nbd ** Tags added: libvirt lxc nbd -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1491563 Title: Failed to spawn lxc instance due to nbd mount in use or rootfs busy Status in OpenStack Compute (nova): New Bug description: I have a devstack change up which runs tempest with nova + the lxc backend and it fails some instances on boot because the nbd device is in use or the rootfs mount is busy: https://review.openstack.org/#/c/219448/ http://logs.openstack.org/48/219448/1/check/gate-tempest-dsvm- full/bc0fc2a/logs/screen-n-cpu.txt.gz?level=TRACE It looks like we're leaking nbd devices and failing to tear down instances properly, e.g.: 2015-09-01 22:10:50.141 ERROR nova.virt.disk.api [req-7e5384a8-34f3-4e3a-98cd-b4bb544804a7 tempest-ServersAdminTestJSON-276960936 tempest-ServersAdminTestJSON-718419646] Failed to mount container filesystem '<nova.virt.disk.api._DiskImage object at 0x7f94ec58cd90>' on '/opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs': -- Failed to mount filesystem: Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd4 /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs Exit code: 32 Stdout: u'' Stderr: u'mount: /dev/nbd4 already mounted or /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs busy\n' 2015-09-01 22:10:50.142 ERROR nova.compute.manager [req-7e5384a8-34f3-4e3a-98cd-b4bb544804a7 tempest-ServersAdminTestJSON-276960936 tempest-ServersAdminTestJSON-718419646] [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Instance failed to spawn 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Traceback (most recent call last): 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/compute/manager.py", line 2138, in _build_resources 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] yield resources 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/compute/manager.py", line 2008, in _build_and_run_instance 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] block_device_info=block_device_info) 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2449, in spawn 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] block_device_info=block_device_info) 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4521, in _create_domain_and_network 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] block_device_info, disk_info): 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] return self.gen.next() 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4430, in _lxc_disk_handler 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] block_device_info, disk_info) 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4382, in _create_domain_setup_lxc 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] container_dir=container_dir) 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] File "/opt/stack/new/nova/nova/virt/disk/api.py", line 452, in setup_container 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] raise exception.NovaException(img.errors) 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] NovaException: 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] -- 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Failed to mount filesystem: Unexpected error while running command. 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd4 /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Exit code: 32 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Stdout: u'' 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] Stderr: u'mount: /dev/nbd4 already mounted or /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs busy\n' 2015-09-01 22:10:50.142 4548 ERROR nova.compute.manager [instance: a9317161-ad31-4948-9c2d-3d4899718a78] 2015-09-01 22:10:50.025 DEBUG nova.virt.disk.mount.api [req-7e5384a8-34f3-4e3a-98cd-b4bb544804a7 tempest-ServersAdminTestJSON-276960936 tempest-ServersAdminTestJSON-718419646] Failed to mount filesystem: Unexpected error while running command. Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd4 /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs Exit code: 32 Stdout: u'' Stderr: u'mount: /dev/nbd4 already mounted or /opt/stack/data/nova/instances/a9317161-ad31-4948-9c2d-3d4899718a78/rootfs busy\n' mnt_dev /opt/stack/new/nova/nova/virt/disk/mount/api.py:237 etc To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1491563/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
