Public bug reported: Creating a fresh bug for this as we haven't seen it in a while since the libvirt event rewrite landed but it did just cause a gate failure...
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5d6/801990/6/gate/tempest- integrated-compute/5d6a7ec/testr_results.html 13 Aug 05 16:01:08.964854 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] Waiting for libvirt event about the detach of device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f is timed out. 47114 Aug 05 16:01:08.969793 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: DEBUG nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] Failed to detach device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Libvirt did not re port any error but the device is still in the config. {{(pid=104299) _detach_from_live_with_retry /opt/stack/nova/nova/virt/libvirt/driver.py:2394}} 47115 Aug 05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. De vice is still attached to the guest. 47116 Aug 05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: WARNING nova.virt.block_device [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] [instance: ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Devi ce detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. 47117 Aug 05 16:01:09.024848 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: DEBUG oslo_concurrency.lockutils [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tem pest-AttachVolumeNegativeTest-1260221999-project] Lock "ce4b444f-1587-43e9-b708-d2ae2c9cd59f" released by "nova.compute.manager.ComputeManager.detach_volume.<locals>.do_detach_volume" :: held 201.180s {{(p id=104299) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:367}} 47118 Aug 05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 temp est-AttachVolumeNegativeTest-1260221999-project] Exception during message handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. $ logsearch storedsearch bug-xxx-detach Running stored search: bug-xxx-detach: branches: - master files: - controller/logs/screen-n-cpu.txt job-groups: - nova-devstack limit: 100 project: openstack/nova regex: from the live domain config. Device is still attached to the guest. result: FAILURE [..] 5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47115:Aug 05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. 5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47116:Aug 05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: WARNING nova.virt.block_device [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] [instance: ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. 5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47118:Aug 05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] Exception during message handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. 5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47170:Aug 05 16:01:09.101298 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. Builds with matching logs 1/100: +----------------------------------+---------------------+-----------------------------------+----------+----------------------------+ | uuid | finished | review | pipeline | job | +----------------------------------+---------------------+-----------------------------------+----------+----------------------------+ | 5d6a7ec36f104f10a1f49fe44909aa23 | 2021-08-05T16:25:36 | https://review.opendev.org/801990 | gate | tempest-integrated-compute | +----------------------------------+---------------------+-----------------------------------+----------+----------------------------+ ** Affects: nova Importance: Undecided Status: New ** Tags: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1939108 Title: tempest.api.compute.volumes.test_attach_volume_negative.AttachVolumeNegativeTest fails due to detach failure during cleanup Status in OpenStack Compute (nova): New Bug description: Creating a fresh bug for this as we haven't seen it in a while since the libvirt event rewrite landed but it did just cause a gate failure... https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5d6/801990/6/gate/tempest- integrated-compute/5d6a7ec/testr_results.html 13 Aug 05 16:01:08.964854 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] Waiting for libvirt event about the detach of device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f is timed out. 47114 Aug 05 16:01:08.969793 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: DEBUG nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] Failed to detach device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Libvirt did not re port any error but the device is still in the config. {{(pid=104299) _detach_from_live_with_retry /opt/stack/nova/nova/virt/libvirt/driver.py:2394}} 47115 Aug 05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. De vice is still attached to the guest. 47116 Aug 05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: WARNING nova.virt.block_device [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe st-AttachVolumeNegativeTest-1260221999-project] [instance: ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Devi ce detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. 47117 Aug 05 16:01:09.024848 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: DEBUG oslo_concurrency.lockutils [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tem pest-AttachVolumeNegativeTest-1260221999-project] Lock "ce4b444f-1587-43e9-b708-d2ae2c9cd59f" released by "nova.compute.manager.ComputeManager.detach_volume.<locals>.do_detach_volume" :: held 201.180s {{(p id=104299) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:367}} 47118 Aug 05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 temp est-AttachVolumeNegativeTest-1260221999-project] Exception during message handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. $ logsearch storedsearch bug-xxx-detach Running stored search: bug-xxx-detach: branches: - master files: - controller/logs/screen-n-cpu.txt job-groups: - nova-devstack limit: 100 project: openstack/nova regex: from the live domain config. Device is still attached to the guest. result: FAILURE [..] 5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47115:Aug 05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. 5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47116:Aug 05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: WARNING nova.virt.block_device [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] [instance: ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. 5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47118:Aug 05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] Exception during message handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. 5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47170:Aug 05 16:01:09.101298 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest. Builds with matching logs 1/100: +----------------------------------+---------------------+-----------------------------------+----------+----------------------------+ | uuid | finished | review | pipeline | job | +----------------------------------+---------------------+-----------------------------------+----------+----------------------------+ | 5d6a7ec36f104f10a1f49fe44909aa23 | 2021-08-05T16:25:36 | https://review.opendev.org/801990 | gate | tempest-integrated-compute | +----------------------------------+---------------------+-----------------------------------+----------+----------------------------+ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1939108/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
