Public bug reported:
This happens with OpenStack Newton.
We use some outdated openstack clients, that do not provide the "floating ip
delete" command. Therefore we use the "ip floating delete" command.
The floating IP is allocated using the "ip floating create" command. The
floating IP is then associated to a newly started virtual machine.
After some automated tests, the command to delete the virtual machine is sent.
To keep the project clean, the floating IP is deleted afterwards using the "ip
floating delete" command. This command fails from time to time due to some race
condition.
This race condition seems to be the following. Nova API requests
information about the floating IP from neutron:
GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gAAAAABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"floatingip": {"router_id": "aaab3136-d551-4e23-a62a-d16f89d34ec5",
"status": "ACTIVE", "description": "", "updated_at":
"2017-07-18T03:37:33Z", "dns_domain": "", "floating_network_id":
"f3d64d76-a4e5-473b-878f-78b032ab89df", "fixed_ip_address":
"192.168.3.10", "floating_ip_address": "10.50.0.62", "revision_number":
2, "port_id": "4b082eae-a527-45e6-8603-0d0882098e39", "id": "3aeb2a7e-
ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at":
"2017-07-18T03:37:04Z", "tenant_id": "7829521236b143d2a6778e09ba588ec0",
"project_id": "7829521236b143d2a6778e09ba588ec0"}}
In the response the floating IP still seems to be associated to the
virtual machine - but the request to delete the virtual machine was
already sent.
Nova API requests information about the port, too:
GET /v2.0/ports/4b082eae-a527-45e6-8603-0d0882098e39.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gAAAAABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"NeutronError": {"message": "Port 4b082eae-a527-45e6-8603-0d0882098e39
could not be found.", "type": "PortNotFound", "detail": ""}}
But the virtual machine and its floating IP association seems to be
deleted in the meantime. This makes Nova API fail to delete the floating
IP:
2017-07-18 05:38:41.798 7170 INFO nova.api.openstack.wsgi
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] HTTP exception thrown: Floating IP not found
for ID 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 05:38:41.800 7170 INFO nova.osapi_compute.wsgi.server
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 404 len: 466 time: 0.2575810
openstack ip floating delete 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 11:32:50.867 7171 INFO nova.osapi_compute.wsgi.server
[req-dafa9bec-fe87-47f6-8c44-4420b074da4d
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 200 len: 475 time: 0.2409530
2017-07-18 11:32:51.654 7171 INFO nova.osapi_compute.wsgi.server
[req-8f93afcb-40ff-4d8e-9f2f-c6fa860a8931
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "DELETE
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 202 len: 337 time: 0.7844548
GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gAAAAABZbdXC9iHhy48rN_gmdR8mrhlJy9d5sLLMQS6Vj79oxcaa23yX6xzKzKix9Fqbd7vo-5jH_JW1IzaCDxp7HO0tIfHX1sc1-I0vLCZq7_w5_Ga50-k1Ra_-9_adi1GzqxKRIjE7Gfp_Fe7z4pIn1XNgur99McLgBaoBP4cyJ0nPoC05p0QSV1oMj8Yjb6dNrdCtmjSQNt_Els7ELq_9SMaSVN3zGi4BLlddAY9TcDqTop0yu-A
{"floatingip": {"router_id": null, "status": "DOWN", "description": "",
"updated_at": "2017-07-18T03:38:41Z", "dns_domain": "",
"floating_network_id": "f3d64d76-a4e5-473b-878f-78b032ab89df",
"fixed_ip_address": null, "floating_ip_address": "10.50.0.62",
"revision_number": 3, "port_id": null, "id": "3aeb2a7e-
ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at":
"2017-07-18T03:37:04Z", "tenant_id": "7829521236b143d2a6778e09ba588ec0",
"project_id": "7829521236b143d2a6778e09ba588ec0"}}
- some other requests to networks and the IP address 10.50.0.62
DELETE /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gAAAAABZbdXC9iHhy48rN_gmdR8mrhlJy9d5sLLMQS6Vj79oxcaa23yX6xzKzKix9Fqbd7vo-5jH_JW1IzaCDxp7HO0tIfHX1sc1-I0vLCZq7_w5_Ga50-k1Ra_-9_adi1GzqxKRIjE7Gfp_Fe7z4pIn1XNgur99McLgBaoBP4cyJ0nPoC05p0QSV1oMj8Yjb6dNrdCtmjSQNt_Els7ELq_9SMaSVN3zGi4BLlddAY9TcDqTop0yu-A
Content-Length: 0
HTTP/1.1 204 No Content
Content-Length: 0
X-Openstack-Request-Id: req-5aee2081-b069-42c7-9c21-57b58bcb6fc5
Date: Tue, 18 Jul 2017 09:32:51 GMT
Connection: keep-alive
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1704975
Title:
Nova API floating IP delete fails
Status in OpenStack Compute (nova):
New
Bug description:
This happens with OpenStack Newton.
We use some outdated openstack clients, that do not provide the "floating ip
delete" command. Therefore we use the "ip floating delete" command.
The floating IP is allocated using the "ip floating create" command. The
floating IP is then associated to a newly started virtual machine.
After some automated tests, the command to delete the virtual machine is
sent. To keep the project clean, the floating IP is deleted afterwards using
the "ip floating delete" command. This command fails from time to time due to
some race condition.
This race condition seems to be the following. Nova API requests
information about the floating IP from neutron:
GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gAAAAABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"floatingip": {"router_id": "aaab3136-d551-4e23-a62a-d16f89d34ec5",
"status": "ACTIVE", "description": "", "updated_at":
"2017-07-18T03:37:33Z", "dns_domain": "", "floating_network_id":
"f3d64d76-a4e5-473b-878f-78b032ab89df", "fixed_ip_address":
"192.168.3.10", "floating_ip_address": "10.50.0.62",
"revision_number": 2, "port_id":
"4b082eae-a527-45e6-8603-0d0882098e39", "id": "3aeb2a7e-
ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at":
"2017-07-18T03:37:04Z", "tenant_id":
"7829521236b143d2a6778e09ba588ec0", "project_id":
"7829521236b143d2a6778e09ba588ec0"}}
In the response the floating IP still seems to be associated to the
virtual machine - but the request to delete the virtual machine was
already sent.
Nova API requests information about the port, too:
GET /v2.0/ports/4b082eae-a527-45e6-8603-0d0882098e39.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gAAAAABZbYLBVmPuNg6lSJzyQEuUbSfAcUgJTWDYrYaRJM2ZVzfRSpKPINNfl61M7Ohdfm-1lfx03_RqXhCGBRUKrPYgBpeKWCmiF-hsd2CActr6LMw3dbOHPrrPl8JOvVQ36caRyiDSFa4xY2getQjAfitJgdQknspRUdzpJAU8jvPxxHPSXvfrAWM8J1M2NzDnJKs0-JnZmYK5NlbJDcRMgfLtknzTGJJs6TnhqaDts_i234RsWf8
{"NeutronError": {"message": "Port
4b082eae-a527-45e6-8603-0d0882098e39 could not be found.", "type":
"PortNotFound", "detail": ""}}
But the virtual machine and its floating IP association seems to be
deleted in the meantime. This makes Nova API fail to delete the
floating IP:
2017-07-18 05:38:41.798 7170 INFO nova.api.openstack.wsgi
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] HTTP exception thrown: Floating IP not found
for ID 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 05:38:41.800 7170 INFO nova.osapi_compute.wsgi.server
[req-b8f7414b-6398-4738-b660-fac693a187ab
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 404 len: 466 time: 0.2575810
openstack ip floating delete 3aeb2a7e-ac37-4712-ac69-80dc83a060e6
2017-07-18 11:32:50.867 7171 INFO nova.osapi_compute.wsgi.server
[req-dafa9bec-fe87-47f6-8c44-4420b074da4d
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "GET
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 200 len: 475 time: 0.2409530
2017-07-18 11:32:51.654 7171 INFO nova.osapi_compute.wsgi.server
[req-8f93afcb-40ff-4d8e-9f2f-c6fa860a8931
807ca03b46124ca28309d06a8e66d25e7b7adfe872a6f58903552439304fa173
7829521236b143d2a6778e09ba588ec0 - 27a851d6a3de43b8a4a4900dfa0c3141
27a851d6a3de43b8a4a4900dfa0c3141] 10.20.30.237 "DELETE
/v2.1/7829521236b143d2a6778e09ba588ec0/os-floating-ips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6
HTTP/1.1" status: 202 len: 337 time: 0.7844548
GET /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gAAAAABZbdXC9iHhy48rN_gmdR8mrhlJy9d5sLLMQS6Vj79oxcaa23yX6xzKzKix9Fqbd7vo-5jH_JW1IzaCDxp7HO0tIfHX1sc1-I0vLCZq7_w5_Ga50-k1Ra_-9_adi1GzqxKRIjE7Gfp_Fe7z4pIn1XNgur99McLgBaoBP4cyJ0nPoC05p0QSV1oMj8Yjb6dNrdCtmjSQNt_Els7ELq_9SMaSVN3zGi4BLlddAY9TcDqTop0yu-A
{"floatingip": {"router_id": null, "status": "DOWN", "description":
"", "updated_at": "2017-07-18T03:38:41Z", "dns_domain": "",
"floating_network_id": "f3d64d76-a4e5-473b-878f-78b032ab89df",
"fixed_ip_address": null, "floating_ip_address": "10.50.0.62",
"revision_number": 3, "port_id": null, "id": "3aeb2a7e-
ac37-4712-ac69-80dc83a060e6", "dns_name": "", "created_at":
"2017-07-18T03:37:04Z", "tenant_id":
"7829521236b143d2a6778e09ba588ec0", "project_id":
"7829521236b143d2a6778e09ba588ec0"}}
- some other requests to networks and the IP address 10.50.0.62
DELETE /v2.0/floatingips/3aeb2a7e-ac37-4712-ac69-80dc83a060e6.json HTTP/1.1
Host: controller:9696
Connection: keep-alive
Accept-Encoding: gzip, deflate
Accept: application/json
User-Agent: python-neutronclient
X-Auth-Token:
gAAAAABZbdXC9iHhy48rN_gmdR8mrhlJy9d5sLLMQS6Vj79oxcaa23yX6xzKzKix9Fqbd7vo-5jH_JW1IzaCDxp7HO0tIfHX1sc1-I0vLCZq7_w5_Ga50-k1Ra_-9_adi1GzqxKRIjE7Gfp_Fe7z4pIn1XNgur99McLgBaoBP4cyJ0nPoC05p0QSV1oMj8Yjb6dNrdCtmjSQNt_Els7ELq_9SMaSVN3zGi4BLlddAY9TcDqTop0yu-A
Content-Length: 0
HTTP/1.1 204 No Content
Content-Length: 0
X-Openstack-Request-Id: req-5aee2081-b069-42c7-9c21-57b58bcb6fc5
Date: Tue, 18 Jul 2017 09:32:51 GMT
Connection: keep-alive
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1704975/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
