Public bug reported: Ran into this with the new nova-multi-cell job:
http://logs.openstack.org/22/655222/3/check/nova-multi- cell/c1ba7e0/controller/logs/screen-n-api.txt.gz#_Apr_25_20_35_15_249786 Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips [None req-3b6da167-cd81-4f88-a27b-05876cf4e6f4 tempest-FloatingIPsAssociationTestJSON-110201909 tempest-FloatingIPsAssociationTestJSON-110201909] Unable to associate floating IP 172.24.5.15 to fixed IP 10.1.0.10 for instance be0ea845-25df-4528-8fcf-c27835f41636. Error: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.: nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found. Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips Traceback (most recent call last): Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/api/openstack/compute/floating_ips.py", line 266, in _add_floating_ip Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips fixed_address=fixed_address) Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/network/base_api.py", line 83, in wrapper Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips res = f(self, context, *args, **kwargs) Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2401, in associate_floating_ip Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips orig_instance_uuid) Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips result = fn(cls, context, *args, **kwargs) Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/objects/instance.py", line 505, in get_by_uuid Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips use_slave=use_slave) Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 213, in wrapper Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips return f(*args, **kwargs) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/objects/instance.py", line 497, in _db_instance_get_by_uuid Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips columns_to_join=columns_to_join) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/api.py", line 758, in instance_get_by_uuid Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips return IMPL.instance_get_by_uuid(context, uuid, columns_to_join) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 171, in wrapper Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips return f(*args, **kwargs) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 258, in wrapped Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips return f(context, *args, **kwargs) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1827, in instance_get_by_uuid Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips columns_to_join=columns_to_join) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1836, in _instance_get_by_uuid Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips raise exception.InstanceNotFound(instance_id=uuid) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found. Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips Due to this old code: https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/network/neutronv2/api.py#L2400 https://review.opendev.org/#/c/33054/ The idea there is if you're associating a floating IP from one server to another, we try to refresh the network info cache on the instance that previously had the floating IP so it's not stale and you have 2 instances reporting they have the same floating IP until the cache is cleaned up. The problem is in a multi-cell environment the servers could be in different cells and the context is targeted at the cell we're assigning the floating IP *to* but will fail to lookup the instance in the other cell that the floating IP is coming *from* (the original instance). This results in a 400 error in the API (I'm surprised it's not returning a 404 or 500 actually): https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/api/openstack/compute/floating_ips.py#L282 ** Affects: nova Importance: Medium Status: Triaged ** Tags: api cells neutron -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1826472 Title: addFloatingIp action can fail if re-associating a floating IP to an instance in another cell Status in OpenStack Compute (nova): Triaged Bug description: Ran into this with the new nova-multi-cell job: http://logs.openstack.org/22/655222/3/check/nova-multi- cell/c1ba7e0/controller/logs/screen-n-api.txt.gz#_Apr_25_20_35_15_249786 Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips [None req-3b6da167-cd81-4f88-a27b-05876cf4e6f4 tempest-FloatingIPsAssociationTestJSON-110201909 tempest-FloatingIPsAssociationTestJSON-110201909] Unable to associate floating IP 172.24.5.15 to fixed IP 10.1.0.10 for instance be0ea845-25df-4528-8fcf-c27835f41636. Error: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found.: nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found. Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips Traceback (most recent call last): Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/api/openstack/compute/floating_ips.py", line 266, in _add_floating_ip Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips fixed_address=fixed_address) Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/network/base_api.py", line 83, in wrapper Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips res = f(self, context, *args, **kwargs) Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/network/neutronv2/api.py", line 2401, in associate_floating_ip Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips orig_instance_uuid) Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips result = fn(cls, context, *args, **kwargs) Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/objects/instance.py", line 505, in get_by_uuid Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips use_slave=use_slave) Apr 25 20:35:15.249786 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 213, in wrapper Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips return f(*args, **kwargs) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/objects/instance.py", line 497, in _db_instance_get_by_uuid Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips columns_to_join=columns_to_join) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/api.py", line 758, in instance_get_by_uuid Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips return IMPL.instance_get_by_uuid(context, uuid, columns_to_join) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 171, in wrapper Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips return f(*args, **kwargs) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 258, in wrapped Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips return f(context, *args, **kwargs) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1827, in instance_get_by_uuid Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips columns_to_join=columns_to_join) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1836, in _instance_get_by_uuid Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips raise exception.InstanceNotFound(instance_id=uuid) Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips nova.exception.InstanceNotFound: Instance 85ce6664-7b76-460a-88fe-5fd05b7c34bc could not be found. Apr 25 20:35:15.251179 ubuntu-bionic-rax-ord-0005726254 [email protected][19571]: ERROR nova.api.openstack.compute.floating_ips Due to this old code: https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/network/neutronv2/api.py#L2400 https://review.opendev.org/#/c/33054/ The idea there is if you're associating a floating IP from one server to another, we try to refresh the network info cache on the instance that previously had the floating IP so it's not stale and you have 2 instances reporting they have the same floating IP until the cache is cleaned up. The problem is in a multi-cell environment the servers could be in different cells and the context is targeted at the cell we're assigning the floating IP *to* but will fail to lookup the instance in the other cell that the floating IP is coming *from* (the original instance). This results in a 400 error in the API (I'm surprised it's not returning a 404 or 500 actually): https://github.com/openstack/nova/blob/a991980863f056323c1ee9fd6a46dbc4cb899eca/nova/api/openstack/compute/floating_ips.py#L282 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1826472/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
