Public bug reported: stable/victoria merge process is effectively blocked because test_live_block_migration_with_attached_volume tempest test (called as tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration_with_attached_volume and tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration_with_attached_volume) is failing sporadically. As a result, nova-grenade-multinode or nova- live-migration are failing sporadically: one of these jobs fails almost every time.
I have used failed tests for review https://review.opendev.org/c/openstack/nova/+/833436 to analyze the problem. Picture is always the same: 1. Tempest deletes volume attachment [1] 2. nova-compute fails to detach volume because of error [2] It looks like a communication problem between nova-compute and libvirt. [1] 2023-02-28 12:43:25.610324 | controller | 2023-02-28 12:40:08,991 6532 INFO [tempest.lib.common.rest_client] Request (LiveAutoBlockMigrationV225Test:_run_cleanups): 202 DELETE http://213.32.76.87/compute/v2.1/servers/e49ccfac-97fc-4b8d-a15a-6a319d8c9a9b/os-volume_attachments/e0d103c8-cc39-4ccc-a91e-a415ff68bdee 0.217s [2] Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 472, in _do_wait_and_retry_detach Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server device=alternative_device_name, reason=reason) Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for vdb: Unable to detach the device from the live config. ** Affects: nova Importance: Undecided Status: New ** Tags: testing -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2009156 Title: stable/victoria tests are very unstable because of tempest test_live_block_migration_with_attached_volume Status in OpenStack Compute (nova): New Bug description: stable/victoria merge process is effectively blocked because test_live_block_migration_with_attached_volume tempest test (called as tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_live_block_migration_with_attached_volume and tempest.api.compute.admin.test_live_migration.LiveAutoBlockMigrationV225Test.test_live_block_migration_with_attached_volume) is failing sporadically. As a result, nova-grenade-multinode or nova- live-migration are failing sporadically: one of these jobs fails almost every time. I have used failed tests for review https://review.opendev.org/c/openstack/nova/+/833436 to analyze the problem. Picture is always the same: 1. Tempest deletes volume attachment [1] 2. nova-compute fails to detach volume because of error [2] It looks like a communication problem between nova-compute and libvirt. [1] 2023-02-28 12:43:25.610324 | controller | 2023-02-28 12:40:08,991 6532 INFO [tempest.lib.common.rest_client] Request (LiveAutoBlockMigrationV225Test:_run_cleanups): 202 DELETE http://213.32.76.87/compute/v2.1/servers/e49ccfac-97fc-4b8d-a15a-6a319d8c9a9b/os-volume_attachments/e0d103c8-cc39-4ccc-a91e-a415ff68bdee 0.217s [2] Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server File "/opt/stack/old/nova/nova/virt/libvirt/guest.py", line 472, in _do_wait_and_retry_detach Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server device=alternative_device_name, reason=reason) Feb 28 12:45:26.707716 np0033291896 nova-compute[15752]: ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for vdb: Unable to detach the device from the live config. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2009156/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
