Public bug reported: Description ===========
When rabbitMQ unstable, there might be a chance that method https://github.com/openstack/nova/blob/7a1222a8654684262a8e589d91e67f2b9a9da336/nova/compute/api.py#L4741 will timeout but bdm is successfully created. Under such cases, volume will be shown in server show, but cannot be detached. and volume status is available. Steps to reproduce ================== there might be no way to safely reproduce this failure, because when rabbitmq is unstable, many other services will also show unusual behavior. Expected result =============== We should be able to remove such attachment from api without manually fixing db... ```console root@mgt02:~# openstack server show 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e +-------------------------------------+-------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------------------+-------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | cn-north-3a | | OS-EXT-SRV-ATTR:host | compute01 | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute01 | | OS-EXT-SRV-ATTR:instance_name | instance-0000ce4c | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2021-03-29T09:06:38.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | newsql-net=192.168.1.217; service_mgt=100.114.3.41 | | config_drive | True | | created | 2021-03-29T09:05:19Z | | flavor | newsql_2C8G40G_general (51db3192-cece-4b9a-9969-7916b4543beb) | | hostId | cf1f3937a3286677b3020d817541ac33d7c8f1ca74be49b26f128093 | | id | 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e | | image | newsql-bini2.0.0alpha-ubuntu18.04-x64-20210112-pub (4531e3bf-0433-40c6-816b-6763f9d02c7a) | | key_name | None | | name | NewSQL-1abc5b28-b9e6-45cd-893d-5bb3a7732a43-3 | | progress | 0 | | project_id | acfcc87fc1db430880f0cb1cce410906 | | properties | productTag='NewSQL' | | security_groups | name='default' | | | name='csf-NewSQL-cluster-security-group' | | status | ACTIVE | | updated | 2021-03-29T09:06:39Z | | user_id | a38ef24677cc4a45a143a31c5fb59ee9 | | volumes_attached | id='1351f861-55a7-4dc5-bf5c-31963657876d' | +-------------------------------------+-------------------------------------------------------------------------------------------+ root@mgt02:~# openstack server remove volume 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e 1351f861-55a7-4dc5-bf5c-31963657876d Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-2ca68e00-760d-47c2-913e-b0fe54e86296) (HTTP 400) (Request-ID: req-264aea62-a970-4241-ab14-ea7aad74cd6e) ``` Actual result ============= openstack server remove volume failed Environment =========== 1. stable/victoria in k8s 2. qemu+kvm 2. ceph rbd 3. neutron ml2 openvswitch ** Affects: nova Importance: Undecided Status: New ** Description changed: Description =========== When rabbitMQ unstable, there might be a chance that method https://github.com/openstack/nova/blob/7a1222a8654684262a8e589d91e67f2b9a9da336/nova/compute/api.py#L4741 will timeout but bdm is successfully created. - Under such cases, such volume will be shown in server show, but cannot - be detached. + Under such cases, volume will be shown in server show, but cannot be + detached. and volume status is available. Steps to reproduce ================== there might be no way to safely reproduce this failure, because when rabbitmq is unstable, many other services will also show unusual behavior. Expected result =============== We should be able to remove such attachment from api without manually fixing db... ```console root@mgt02:~# openstack server show 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e +-------------------------------------+-------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------------------+-------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | cn-north-3a | | OS-EXT-SRV-ATTR:host | compute01 | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute01 | | OS-EXT-SRV-ATTR:instance_name | instance-0000ce4c | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2021-03-29T09:06:38.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | newsql-net=192.168.1.217; service_mgt=100.114.3.41 | | config_drive | True | | created | 2021-03-29T09:05:19Z | | flavor | newsql_2C8G40G_general (51db3192-cece-4b9a-9969-7916b4543beb) | | hostId | cf1f3937a3286677b3020d817541ac33d7c8f1ca74be49b26f128093 | | id | 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e | | image | newsql-bini2.0.0alpha-ubuntu18.04-x64-20210112-pub (4531e3bf-0433-40c6-816b-6763f9d02c7a) | | key_name | None | | name | NewSQL-1abc5b28-b9e6-45cd-893d-5bb3a7732a43-3 | | progress | 0 | | project_id | acfcc87fc1db430880f0cb1cce410906 | | properties | productTag='NewSQL' | | security_groups | name='default' | | | name='csf-NewSQL-cluster-security-group' | | status | ACTIVE | | updated | 2021-03-29T09:06:39Z | | user_id | a38ef24677cc4a45a143a31c5fb59ee9 | | volumes_attached | id='1351f861-55a7-4dc5-bf5c-31963657876d' | +-------------------------------------+-------------------------------------------------------------------------------------------+ root@mgt02:~# openstack server remove volume 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e 1351f861-55a7-4dc5-bf5c-31963657876d Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-2ca68e00-760d-47c2-913e-b0fe54e86296) (HTTP 400) (Request-ID: req-264aea62-a970-4241-ab14-ea7aad74cd6e) ``` Actual result ============= openstack server remove volume failed Environment =========== 1. stable/victoria in k8s 2. qemu+kvm 2. ceph rbd 3. neutron ml2 openvswitch -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1921804 Title: leftover bdm when rabbitmq unstable Status in OpenStack Compute (nova): New Bug description: Description =========== When rabbitMQ unstable, there might be a chance that method https://github.com/openstack/nova/blob/7a1222a8654684262a8e589d91e67f2b9a9da336/nova/compute/api.py#L4741 will timeout but bdm is successfully created. Under such cases, volume will be shown in server show, but cannot be detached. and volume status is available. Steps to reproduce ================== there might be no way to safely reproduce this failure, because when rabbitmq is unstable, many other services will also show unusual behavior. Expected result =============== We should be able to remove such attachment from api without manually fixing db... ```console root@mgt02:~# openstack server show 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e +-------------------------------------+-------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------------------+-------------------------------------------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | cn-north-3a | | OS-EXT-SRV-ATTR:host | compute01 | | OS-EXT-SRV-ATTR:hypervisor_hostname | compute01 | | OS-EXT-SRV-ATTR:instance_name | instance-0000ce4c | | OS-EXT-STS:power_state | Running | | OS-EXT-STS:task_state | None | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2021-03-29T09:06:38.000000 | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | newsql-net=192.168.1.217; service_mgt=100.114.3.41 | | config_drive | True | | created | 2021-03-29T09:05:19Z | | flavor | newsql_2C8G40G_general (51db3192-cece-4b9a-9969-7916b4543beb) | | hostId | cf1f3937a3286677b3020d817541ac33d7c8f1ca74be49b26f128093 | | id | 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e | | image | newsql-bini2.0.0alpha-ubuntu18.04-x64-20210112-pub (4531e3bf-0433-40c6-816b-6763f9d02c7a) | | key_name | None | | name | NewSQL-1abc5b28-b9e6-45cd-893d-5bb3a7732a43-3 | | progress | 0 | | project_id | acfcc87fc1db430880f0cb1cce410906 | | properties | productTag='NewSQL' | | security_groups | name='default' | | | name='csf-NewSQL-cluster-security-group' | | status | ACTIVE | | updated | 2021-03-29T09:06:39Z | | user_id | a38ef24677cc4a45a143a31c5fb59ee9 | | volumes_attached | id='1351f861-55a7-4dc5-bf5c-31963657876d' | +-------------------------------------+-------------------------------------------------------------------------------------------+ root@mgt02:~# openstack server remove volume 4e5c3c7d-6b4c-4841-9e6e-9a3374036a3e 1351f861-55a7-4dc5-bf5c-31963657876d Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-2ca68e00-760d-47c2-913e-b0fe54e86296) (HTTP 400) (Request-ID: req-264aea62-a970-4241-ab14-ea7aad74cd6e) ``` Actual result ============= openstack server remove volume failed Environment =========== 1. stable/victoria in k8s 2. qemu+kvm 2. ceph rbd 3. neutron ml2 openvswitch To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1921804/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
