Public bug reported:
Description
===========
Extending volume in-use may fail, in that case, CLI "nova instance-action-list"
should show ERROR in the coulume of message.However , when we catch exceptions
in /nova/compute/manager.py
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8659-L8677
nova extends volume will fail ,however CLI "nova instance-action vmid req-id"
returns success,that's wrong
===========
1.create volume-backend VM
compute]# nova boot --flavor 1 --nic
net-id=27105f4f-5847-4538-862d-6565ad002c9e --boot-volume
53bc2318-ae6c-42b5-ad3b-f197f405cf1e VM
+--------------------------------------+-------------------------------------------------+
| Property | Value
|
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL
|
| OS-EXT-AZ:availability_zone |
|
| OS-EXT-SRV-ATTR:host | -
|
| OS-EXT-SRV-ATTR:hostname | vm
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | -
|
| OS-EXT-SRV-ATTR:instance_name |
|
| OS-EXT-SRV-ATTR:kernel_id |
|
| OS-EXT-SRV-ATTR:launch_index | 0
|
| OS-EXT-SRV-ATTR:ramdisk_id |
|
| OS-EXT-SRV-ATTR:reservation_id | r-1jrun74v
|
| OS-EXT-SRV-ATTR:root_device_name | -
|
| OS-EXT-SRV-ATTR:user_data | -
|
| OS-EXT-STS:power_state | 0
|
| OS-EXT-STS:task_state | scheduling
|
| OS-EXT-STS:vm_state | building
|
| OS-SRV-USG:launched_at | -
|
| OS-SRV-USG:terminated_at | -
|
| accessIPv4 |
|
| accessIPv6 |
|
| adminPass | TbPSQRBic9NQ
|
| config_drive |
|
| created | 2019-09-06T06:41:55Z
|
| description | -
|
| flavor:disk | 1
|
| flavor:ephemeral | 0
|
| flavor:extra_specs | {}
|
| flavor:original_name | m1.tiny
|
| flavor:ram | 512
|
| flavor:swap | 0
|
| flavor:vcpus | 1
|
| hostId |
|
| host_status |
|
| id | fe3c831c-95af-4f0b-b32a-31fb1e425462
|
| image | Attempt to boot from volume - no image
supplied |
| key_name | -
|
| locked | False
|
| locked_reason | -
|
| metadata | {}
|
| name | VM
|
| os-extended-volumes:volumes_attached | []
|
| progress | 0
|
| security_groups | default
|
| server_groups | []
|
| status | BUILD
|
| tags | []
|
| tenant_id | 92cd1d753cf54f3496fda649212cc37b
|
| trusted_image_certificates | -
|
| updated | 2019-09-06T06:41:55Z
|
| user_id | 85c12fb6eada44bc879da10ea394832b
|
+--------------------------------------+-------------------------------------------------+
2.Setting specific scenarios to trigger exceptions in
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8659-L8677
for example :insert 'extended_volume_id=1' in
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8656-L8658
,
pass objects.BlockDeviceMapping.get_by_volume_and_instance invalid
extended_volume_id
3. extend instance's volume
root@controller compute]# cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type |
Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 53bc2318-ae6c-42b5-ad3b-f197f405cf1e | in-use | - | 2 | lvmdriver-1 |
true | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
[root@controller compute]# cinder extend 53bc2318-ae6c-42b5-ad3b-f197f405cf1e 3
[root@controller compute]# cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type |
Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 53bc2318-ae6c-42b5-ad3b-f197f405cf1e | in-use | - | 3 | lvmdriver-1 |
true | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
4.volume extend successfully in cinder , check extending result in nova, it
shows success,In fact,extending volume in nova failed
[root@controller compute]# nova instance-action-list
fe3c831c-95af-4f0b-b32a-31fb1e425462
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
| Action | Request_ID | Message |
Start_Time | Updated_At |
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
| create | req-c3220c13-1de3-435d-a765-c1e931025974 | - |
2019-09-06T06:41:54.000000 | 2019-09-06T06:42:02.000000 |
| extend_volume | req-9c233e67-0105-4a76-86aa-36fdd59593dd | - |
2019-09-06T06:42:42.000000 | 2019-09-06T06:42:42.000000 |
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
[root@controller compute]# nova instance-action
fe3c831c-95af-4f0b-b32a-31fb1e425462 req-9c233e67-0105-4a76-86aa-36fdd59593dd
+---------------+---------------------------------------------------------------------------+
| Property | Value
|
+---------------+---------------------------------------------------------------------------+
| action | extend_volume
|
| events | [{u'event': u'compute_extend_volume',
|
| | u'finish_time': u'2019-09-06T06:42:42.000000',
|
| | u'host': u'controller',
|
| | u'hostId':
u'b7bbc1111bdc01fbf6ea645b2942c2a201d5211cf997f93543ab57a9', |
| | u'result': u'Success',
|
| | u'start_time': u'2019-09-06T06:42:42.000000',
|
| | u'traceback': None}]
|
| instance_uuid | fe3c831c-95af-4f0b-b32a-31fb1e425462
|
| message | -
|
| project_id | ade79e9fbd42422baeefd9e5d4f8a66e
|
| request_id | req-9c233e67-0105-4a76-86aa-36fdd59593dd
|
| start_time | 2019-09-06T06:42:42.000000
|
| updated_at | 2019-09-06T06:42:42.000000
|
| user_id | 647971cd085b4489af7ccc0126bf8c93
|
+---------------+---------------------------------------------------------------------------+
[root@controller compute]# journalctl -a --unit [email protected] |grep
req-9c233e67-0105-4a76-86aa-36fdd59593dd
Logs:
Sep 06 14:42:42 controller nova-compute[13260]: DEBUG nova.compute.manager
[req-83c716d0-6be3-4034-9ed4-9d416d74358b
req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance:
fe3c831c-95af-4f0b-b32a-31fb1e425462] Received event
volume-extended-53bc2318-ae6c-42b5-ad3b-f197f405cf1e {{(pid=13260)
external_instance_event /opt/stack/nova/nova/compute/manager.py:8799}}
Sep 06 14:42:42 controller nova-compute[13260]: DEBUG nova.compute.manager
[req-83c716d0-6be3-4034-9ed4-9d416d74358b
req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance:
fe3c831c-95af-4f0b-b32a-31fb1e425462] Handling volume-extended event for volume
53bc2318-ae6c-42b5-ad3b-f197f405cf1e {{(pid=13260) extend_volume
/opt/stack/nova/nova/compute/manager.py:8644}}
Sep 06 14:42:42 controller nova-compute[13260]: WARNING nova.compute.manager
[req-83c716d0-6be3-4034-9ed4-9d416d74358b
req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance:
fe3c831c-95af-4f0b-b32a-31fb1e425462] Extend volume failed, volume 1 is not
attached to instance.: VolumeBDMNotFound_Remote: No volume Block Device Mapping
with id 1.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1843008
Title:
Nova will not detect when extending volume online fails
Status in OpenStack Compute (nova):
New
Bug description:
Description
===========
Extending volume in-use may fail, in that case, CLI "nova
instance-action-list" should show ERROR in the coulume of message.However ,
when we catch exceptions in /nova/compute/manager.py
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8659-L8677
nova extends volume will fail ,however CLI "nova instance-action vmid req-id"
returns success,that's wrong
===========
1.create volume-backend VM
compute]# nova boot --flavor 1 --nic
net-id=27105f4f-5847-4538-862d-6565ad002c9e --boot-volume
53bc2318-ae6c-42b5-ad3b-f197f405cf1e VM
+--------------------------------------+-------------------------------------------------+
| Property | Value
|
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL
|
| OS-EXT-AZ:availability_zone |
|
| OS-EXT-SRV-ATTR:host | -
|
| OS-EXT-SRV-ATTR:hostname | vm
|
| OS-EXT-SRV-ATTR:hypervisor_hostname | -
|
| OS-EXT-SRV-ATTR:instance_name |
|
| OS-EXT-SRV-ATTR:kernel_id |
|
| OS-EXT-SRV-ATTR:launch_index | 0
|
| OS-EXT-SRV-ATTR:ramdisk_id |
|
| OS-EXT-SRV-ATTR:reservation_id | r-1jrun74v
|
| OS-EXT-SRV-ATTR:root_device_name | -
|
| OS-EXT-SRV-ATTR:user_data | -
|
| OS-EXT-STS:power_state | 0
|
| OS-EXT-STS:task_state | scheduling
|
| OS-EXT-STS:vm_state | building
|
| OS-SRV-USG:launched_at | -
|
| OS-SRV-USG:terminated_at | -
|
| accessIPv4 |
|
| accessIPv6 |
|
| adminPass | TbPSQRBic9NQ
|
| config_drive |
|
| created | 2019-09-06T06:41:55Z
|
| description | -
|
| flavor:disk | 1
|
| flavor:ephemeral | 0
|
| flavor:extra_specs | {}
|
| flavor:original_name | m1.tiny
|
| flavor:ram | 512
|
| flavor:swap | 0
|
| flavor:vcpus | 1
|
| hostId |
|
| host_status |
|
| id | fe3c831c-95af-4f0b-b32a-31fb1e425462
|
| image | Attempt to boot from volume - no
image supplied |
| key_name | -
|
| locked | False
|
| locked_reason | -
|
| metadata | {}
|
| name | VM
|
| os-extended-volumes:volumes_attached | []
|
| progress | 0
|
| security_groups | default
|
| server_groups | []
|
| status | BUILD
|
| tags | []
|
| tenant_id | 92cd1d753cf54f3496fda649212cc37b
|
| trusted_image_certificates | -
|
| updated | 2019-09-06T06:41:55Z
|
| user_id | 85c12fb6eada44bc879da10ea394832b
|
+--------------------------------------+-------------------------------------------------+
2.Setting specific scenarios to trigger exceptions in
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8659-L8677
for example :insert 'extended_volume_id=1' in
https://github.com/openstack/nova/blob/7020196aaa2fedd537806fe229e237c91e4f0ca5/nova/compute/manager.py#L8656-L8658
,
pass objects.BlockDeviceMapping.get_by_volume_and_instance invalid
extended_volume_id
3. extend instance's volume
root@controller compute]# cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type |
Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 53bc2318-ae6c-42b5-ad3b-f197f405cf1e | in-use | - | 2 | lvmdriver-1 |
true | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
[root@controller compute]# cinder extend 53bc2318-ae6c-42b5-ad3b-f197f405cf1e
3
[root@controller compute]# cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type |
Bootable | Attached to |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 53bc2318-ae6c-42b5-ad3b-f197f405cf1e | in-use | - | 3 | lvmdriver-1 |
true | fe3c831c-95af-4f0b-b32a-31fb1e425462 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
4.volume extend successfully in cinder , check extending result in nova, it
shows success,In fact,extending volume in nova failed
[root@controller compute]# nova instance-action-list
fe3c831c-95af-4f0b-b32a-31fb1e425462
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
| Action | Request_ID | Message |
Start_Time | Updated_At |
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
| create | req-c3220c13-1de3-435d-a765-c1e931025974 | - |
2019-09-06T06:41:54.000000 | 2019-09-06T06:42:02.000000 |
| extend_volume | req-9c233e67-0105-4a76-86aa-36fdd59593dd | - |
2019-09-06T06:42:42.000000 | 2019-09-06T06:42:42.000000 |
+---------------+------------------------------------------+---------+----------------------------+----------------------------+
[root@controller compute]# nova instance-action
fe3c831c-95af-4f0b-b32a-31fb1e425462 req-9c233e67-0105-4a76-86aa-36fdd59593dd
+---------------+---------------------------------------------------------------------------+
| Property | Value
|
+---------------+---------------------------------------------------------------------------+
| action | extend_volume
|
| events | [{u'event': u'compute_extend_volume',
|
| | u'finish_time': u'2019-09-06T06:42:42.000000',
|
| | u'host': u'controller',
|
| | u'hostId':
u'b7bbc1111bdc01fbf6ea645b2942c2a201d5211cf997f93543ab57a9', |
| | u'result': u'Success',
|
| | u'start_time': u'2019-09-06T06:42:42.000000',
|
| | u'traceback': None}]
|
| instance_uuid | fe3c831c-95af-4f0b-b32a-31fb1e425462
|
| message | -
|
| project_id | ade79e9fbd42422baeefd9e5d4f8a66e
|
| request_id | req-9c233e67-0105-4a76-86aa-36fdd59593dd
|
| start_time | 2019-09-06T06:42:42.000000
|
| updated_at | 2019-09-06T06:42:42.000000
|
| user_id | 647971cd085b4489af7ccc0126bf8c93
|
+---------------+---------------------------------------------------------------------------+
[root@controller compute]# journalctl -a --unit [email protected] |grep
req-9c233e67-0105-4a76-86aa-36fdd59593dd
Logs:
Sep 06 14:42:42 controller nova-compute[13260]: DEBUG nova.compute.manager
[req-83c716d0-6be3-4034-9ed4-9d416d74358b
req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance:
fe3c831c-95af-4f0b-b32a-31fb1e425462] Received event
volume-extended-53bc2318-ae6c-42b5-ad3b-f197f405cf1e {{(pid=13260)
external_instance_event /opt/stack/nova/nova/compute/manager.py:8799}}
Sep 06 14:42:42 controller nova-compute[13260]: DEBUG nova.compute.manager
[req-83c716d0-6be3-4034-9ed4-9d416d74358b
req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance:
fe3c831c-95af-4f0b-b32a-31fb1e425462] Handling volume-extended event for volume
53bc2318-ae6c-42b5-ad3b-f197f405cf1e {{(pid=13260) extend_volume
/opt/stack/nova/nova/compute/manager.py:8644}}
Sep 06 14:42:42 controller nova-compute[13260]: WARNING nova.compute.manager
[req-83c716d0-6be3-4034-9ed4-9d416d74358b
req-9c233e67-0105-4a76-86aa-36fdd59593dd service nova] [instance:
fe3c831c-95af-4f0b-b32a-31fb1e425462] Extend volume failed, volume 1 is not
attached to instance.: VolumeBDMNotFound_Remote: No volume Block Device Mapping
with id 1.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1843008/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
