** Also affects: devstack
Importance: Undecided
Status: New
** Changed in: devstack
Assignee: (unassigned) => Lee Yarwood (lyarwood)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1939108
Title:
Failure to detach volume during Tempest test cleanup due to APIC
related kernel panic within the guest OS
Status in devstack:
New
Status in OpenStack Compute (nova):
Confirmed
Bug description:
Creating a fresh bug for this as we haven't seen it in a while since
the libvirt event rewrite landed but it did just cause a gate
failure...
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5d6/801990/6/gate/tempest-
integrated-compute/5d6a7ec/testr_results.html
13 Aug 05 16:01:08.964854 ubuntu-focal-ovh-gra1-0025782813
nova-compute[104299]: ERROR nova.virt.libvirt.driver [None
req-54714613-ee7e-4c9f-8e60-8251a77a526e
tempest-AttachVolumeNegativeTest-1260221999 tempe
st-AttachVolumeNegativeTest-1260221999-project] Waiting for libvirt event about
the detach of device vdb with device alias virtio-disk1 from instance
ce4b444f-1587-43e9-b708-d2ae2c9cd59f is timed out.
47114 Aug 05 16:01:08.969793 ubuntu-focal-ovh-gra1-0025782813
nova-compute[104299]: DEBUG nova.virt.libvirt.driver [None
req-54714613-ee7e-4c9f-8e60-8251a77a526e
tempest-AttachVolumeNegativeTest-1260221999 tempe
st-AttachVolumeNegativeTest-1260221999-project] Failed to detach device vdb
with device alias virtio-disk1 from instance
ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Libvirt did
not re port any error but the device is still in the config.
{{(pid=104299) _detach_from_live_with_retry
/opt/stack/nova/nova/virt/libvirt/driver.py:2394}}
47115 Aug 05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813
nova-compute[104299]: ERROR nova.virt.libvirt.driver [None
req-54714613-ee7e-4c9f-8e60-8251a77a526e
tempest-AttachVolumeNegativeTest-1260221999 tempe
st-AttachVolumeNegativeTest-1260221999-project] Run out of retry while
detaching device vdb with device alias virtio-disk1 from instance
ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. De vice
is still attached to the guest.
47116 Aug 05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813
nova-compute[104299]: WARNING nova.virt.block_device [None
req-54714613-ee7e-4c9f-8e60-8251a77a526e
tempest-AttachVolumeNegativeTest-1260221999 tempe
st-AttachVolumeNegativeTest-1260221999-project] [instance:
ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume
ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Devi
ce detach failed for vdb: Run out of retry while detaching device vdb with
device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f
from the live domain config. Device is still attached to the guest.
47117 Aug 05 16:01:09.024848 ubuntu-focal-ovh-gra1-0025782813
nova-compute[104299]: DEBUG oslo_concurrency.lockutils [None
req-54714613-ee7e-4c9f-8e60-8251a77a526e
tempest-AttachVolumeNegativeTest-1260221999 tem
pest-AttachVolumeNegativeTest-1260221999-project] Lock
"ce4b444f-1587-43e9-b708-d2ae2c9cd59f" released by
"nova.compute.manager.ComputeManager.detach_volume.<locals>.do_detach_volume"
:: held 201.180s {{(p id=104299) inner
/usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:367}}
47118 Aug 05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813
nova-compute[104299]: ERROR oslo_messaging.rpc.server [None
req-54714613-ee7e-4c9f-8e60-8251a77a526e
tempest-AttachVolumeNegativeTest-1260221999 temp
est-AttachVolumeNegativeTest-1260221999-project] Exception during message
handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run
out of retry while detaching device vdb with device alias virtio-disk1
from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config.
Device is still attached to the guest.
$ logsearch storedsearch bug-xxx-detach
Running stored search:
bug-xxx-detach:
branches:
- master
files:
- controller/logs/screen-n-cpu.txt
job-groups:
- nova-devstack
limit: 100
project: openstack/nova
regex: from the live domain config. Device is still attached to the guest.
result: FAILURE
[..]
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47115:Aug
05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]:
ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e
tempest-AttachVolumeNegativeTest-1260221999
tempest-AttachVolumeNegativeTest-1260221999-project] Run out of retry while
detaching device vdb with device alias virtio-disk1 from instance
ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is
still attached to the guest.
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47116:Aug
05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]:
WARNING nova.virt.block_device [None req-54714613-ee7e-4c9f-8e60-8251a77a526e
tempest-AttachVolumeNegativeTest-1260221999
tempest-AttachVolumeNegativeTest-1260221999-project] [instance:
ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume
ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Device
detach failed for vdb: Run out of retry while detaching device vdb with device
alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the
live domain config. Device is still attached to the guest.
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47118:Aug
05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]:
ERROR oslo_messaging.rpc.server [None req-54714613-ee7e-4c9f-8e60-8251a77a526e
tempest-AttachVolumeNegativeTest-1260221999
tempest-AttachVolumeNegativeTest-1260221999-project] Exception during message
handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run
out of retry while detaching device vdb with device alias virtio-disk1 from
instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config.
Device is still attached to the guest.
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47170:Aug
05 16:01:09.101298 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]:
ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device
detach failed for vdb: Run out of retry while detaching device vdb with device
alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the
live domain config. Device is still attached to the guest.
Builds with matching logs 1/100:
+----------------------------------+---------------------+-----------------------------------+----------+----------------------------+
| uuid | finished | review
| pipeline | job |
+----------------------------------+---------------------+-----------------------------------+----------+----------------------------+
| 5d6a7ec36f104f10a1f49fe44909aa23 | 2021-08-05T16:25:36 |
https://review.opendev.org/801990 | gate | tempest-integrated-compute |
+----------------------------------+---------------------+-----------------------------------+----------+----------------------------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/devstack/+bug/1939108/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
