Public bug reported:
It seems that to reproduce the problem we need a multi node deployment
with at least two nova-compute service.
To reproduce it do the following:
1) create a neutron port
2) boot two instances in parallel with that port
Sometimes both instances become ACTIVE in nova which is clearly wrong.
vagrant@controller:~/devstack$ neutron net-list
+--------------------------------------+---------+----------------------------------------------------------+
| id | name | subnets
|
+--------------------------------------+---------+----------------------------------------------------------+
| fc257a00-d3bf-47e6-b91f-e2cef985c414 | public |
a610715c-614f-492c-8810-f51e03c5d383 2001:db8::/64 |
| | |
a923871f-90ad-4354-935d-db24861a5890 172.24.4.0/24 |
| 7a057b12-0e69-4c31-859e-098263abeeba | private |
04f3b138-d7c6-48e1-98e3-7f70eb7ab4fe fda4:10b7:acaa::/64 |
| | |
ee70023c-f273-471a-8b84-cb25bb64fcf9 10.0.0.0/24 |
+--------------------------------------+---------+----------------------------------------------------------+
vagrant@controller:~/devstack$ neutron port-create
7a057b12-0e69-4c31-859e-098263abeeba
Created a new port:
+-----------------------+-------------------------------------------------------------------------------------------------------------+
| Field | Value
|
+-----------------------+-------------------------------------------------------------------------------------------------------------+
| admin_state_up | True
|
| allowed_address_pairs |
|
| binding:host_id |
|
| binding:profile | {}
|
| binding:vif_details | {}
|
| binding:vif_type | unbound
|
| binding:vnic_type | normal
|
| device_id |
|
| device_owner |
|
| fixed_ips | {"subnet_id": "ee70023c-f273-471a-8b84-cb25bb64fcf9",
"ip_address": "10.0.0.4"} |
| | {"subnet_id": "04f3b138-d7c6-48e1-98e3-7f70eb7ab4fe",
"ip_address": "fda4:10b7:acaa:0:f816:3eff:fe0c:285d"} |
| id | f2da8f78-8ae4-49f0-bca0-5820588d33ea
|
| mac_address | fa:16:3e:0c:28:5d
|
| name |
|
| network_id | 7a057b12-0e69-4c31-859e-098263abeeba
|
| port_security_enabled | True
|
| security_groups | 73853e74-a6c7-4b71-ba45-5b82b5e1ad81
|
| status | DOWN
|
| tenant_id | 16f8c1dbfa2d472da0d1335b8a70aee0
|
+-----------------------+-------------------------------------------------------------------------------------------------------------+
vagrant@controller:~/devstack$ nova boot --image cirros-0.3.4-x86_64-uec
--flavor 42 --nic port-id=f2da8f78-8ae4-49f0-bca0-5820588d33ea vm1 & nova boot
--image cirros-0.3.4-x86_64-uec --flavor 42 --nic
port-id=f2da8f78-8ae4-49f0-bca0-5820588d33ea vm2 &
[1] 18785
[2] 18786
vagrant@controller:~/devstack$ nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
| ID | Name | Status | Task State | Power
State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
| 24e67346-18fe-4cfa-8b01-f017c915a111 | vm1 | ACTIVE | - | Running
| private=fda4:10b7:acaa:0:f816:3eff:fe0c:285d, 10.0.0.4 |
| d90292a2-5b8f-4566-a621-defdf7aa0246 | vm2 | ACTIVE | - | Running
| |
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
Based on the code we have an atomicity problem on the neutron REST API:
At
https://github.com/openstack/nova/blob/1db33ca6c248613cc8a76dcbbf78758001ee02d8/nova/network/neutronv2/api.py#L611
nova will check if the device_id of the neutron port are empty or not. Then
nova sets the current instance_uuid to the device_id later at
https://github.com/openstack/nova/blob/1db33ca6c248613cc8a76dcbbf78758001ee02d8/nova/network/neutronv2/api.py#L692
So it is possible that two nova-compute processes check the port status
before one of them sets the device_id, so as a result both nova-compute
will think that the port are free to use and the slower nova-compute
will overwrite the device_id of the port.
** Affects: nova
Importance: Undecided
Assignee: Balazs Gibizer (balazs-gibizer)
Status: New
** Changed in: nova
Assignee: (unassigned) => Balazs Gibizer (balazs-gibizer)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1493714
Title:
nova allows booting two instances with the same neutron port in
parallel
Status in OpenStack Compute (nova):
New
Bug description:
It seems that to reproduce the problem we need a multi node deployment
with at least two nova-compute service.
To reproduce it do the following:
1) create a neutron port
2) boot two instances in parallel with that port
Sometimes both instances become ACTIVE in nova which is clearly wrong.
vagrant@controller:~/devstack$ neutron net-list
+--------------------------------------+---------+----------------------------------------------------------+
| id | name | subnets
|
+--------------------------------------+---------+----------------------------------------------------------+
| fc257a00-d3bf-47e6-b91f-e2cef985c414 | public |
a610715c-614f-492c-8810-f51e03c5d383 2001:db8::/64 |
| | |
a923871f-90ad-4354-935d-db24861a5890 172.24.4.0/24 |
| 7a057b12-0e69-4c31-859e-098263abeeba | private |
04f3b138-d7c6-48e1-98e3-7f70eb7ab4fe fda4:10b7:acaa::/64 |
| | |
ee70023c-f273-471a-8b84-cb25bb64fcf9 10.0.0.0/24 |
+--------------------------------------+---------+----------------------------------------------------------+
vagrant@controller:~/devstack$ neutron port-create
7a057b12-0e69-4c31-859e-098263abeeba
Created a new port:
+-----------------------+-------------------------------------------------------------------------------------------------------------+
| Field | Value
|
+-----------------------+-------------------------------------------------------------------------------------------------------------+
| admin_state_up | True
|
| allowed_address_pairs |
|
| binding:host_id |
|
| binding:profile | {}
|
| binding:vif_details | {}
|
| binding:vif_type | unbound
|
| binding:vnic_type | normal
|
| device_id |
|
| device_owner |
|
| fixed_ips | {"subnet_id":
"ee70023c-f273-471a-8b84-cb25bb64fcf9", "ip_address": "10.0.0.4"}
|
| | {"subnet_id":
"04f3b138-d7c6-48e1-98e3-7f70eb7ab4fe", "ip_address":
"fda4:10b7:acaa:0:f816:3eff:fe0c:285d"} |
| id | f2da8f78-8ae4-49f0-bca0-5820588d33ea
|
| mac_address | fa:16:3e:0c:28:5d
|
| name |
|
| network_id | 7a057b12-0e69-4c31-859e-098263abeeba
|
| port_security_enabled | True
|
| security_groups | 73853e74-a6c7-4b71-ba45-5b82b5e1ad81
|
| status | DOWN
|
| tenant_id | 16f8c1dbfa2d472da0d1335b8a70aee0
|
+-----------------------+-------------------------------------------------------------------------------------------------------------+
vagrant@controller:~/devstack$ nova boot --image cirros-0.3.4-x86_64-uec
--flavor 42 --nic port-id=f2da8f78-8ae4-49f0-bca0-5820588d33ea vm1 & nova boot
--image cirros-0.3.4-x86_64-uec --flavor 42 --nic
port-id=f2da8f78-8ae4-49f0-bca0-5820588d33ea vm2 &
[1] 18785
[2] 18786
vagrant@controller:~/devstack$ nova list
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
| ID | Name | Status | Task State | Power
State | Networks |
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
| 24e67346-18fe-4cfa-8b01-f017c915a111 | vm1 | ACTIVE | - | Running
| private=fda4:10b7:acaa:0:f816:3eff:fe0c:285d, 10.0.0.4 |
| d90292a2-5b8f-4566-a621-defdf7aa0246 | vm2 | ACTIVE | - | Running
| |
+--------------------------------------+------+--------+------------+-------------+--------------------------------------------------------+
Based on the code we have an atomicity problem on the neutron REST API:
At
https://github.com/openstack/nova/blob/1db33ca6c248613cc8a76dcbbf78758001ee02d8/nova/network/neutronv2/api.py#L611
nova will check if the device_id of the neutron port are empty or not. Then
nova sets the current instance_uuid to the device_id later at
https://github.com/openstack/nova/blob/1db33ca6c248613cc8a76dcbbf78758001ee02d8/nova/network/neutronv2/api.py#L692
So it is possible that two nova-compute processes check the port
status before one of them sets the device_id, so as a result both
nova-compute will think that the port are free to use and the slower
nova-compute will overwrite the device_id of the port.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1493714/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
