Reviewed: https://review.openstack.org/373536 Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=873b5ac837220c11e204c6610782f8b86c90bf03 Submitter: Jenkins Branch: master
commit 873b5ac837220c11e204c6610782f8b86c90bf03 Author: Armando Migliaccio <[email protected]> Date: Tue Sep 20 14:23:40 2016 -0700 Retry port update on IpAddressAllocationNotFound If a port update and a subnet delete interleave, there is a chance that the IPAM update operation raises this exception. Rather than throwing that up to the user under some sort of conflict, bubble up a retry instead; that should bring things back to sanity. Closes-bug: #1622616 Change-Id: Ia8cac09349d4cb722737bdf0bec6c54b9e77f31d ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1622616 Title: delete_subnet update_port appears racey with ipam Status in neutron: Fix Released Bug description: failure spotted in a patch on a delete_subnet call: 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource [req-746d769c-2388-48e0-8e09-38e4190e5364 tempest-PortsTestJSON-432635984 -] delete failed: Exception deleting fixed_ip from port 862b5dea-dca2-4669-b280-867175f5f351 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource Traceback (most recent call last): 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource result = method(request=request, **args) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 526, in delete 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return self._delete(request, id, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 87, in wrapped 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise() 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 83, in wrapped 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(*args, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise() 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(*args, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 123, in wrapped 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource traceback.format_exc()) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise() 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 118, in wrapped 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 548, in _delete 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource obj_deleter(request.context, id, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(self, context, *args, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1159, in delete_subnet 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource "port %s"), port_id) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise() 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1151, in delete_subnet 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.update_port(context, port_id, data) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(self, context, *args, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1370, in update_port 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource port) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1183, in update_port 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource new_port=new_port) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_backend_mixin.py", line 724, in update_port 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource new_port.get('mac_address')) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 325, in update_port_with_ips 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource new_mac) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 297, in _update_ips_for_port 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource changes.remove) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 93, in _ipam_deallocate_ips 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource "external system for %s"), addresses) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise() 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 70, in _ipam_deallocate_ips 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource ipam_subnet.deallocate(ip['ip_address']) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 222, in deallocate 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource ip_address=address) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource IpAddressAllocationNotFound: Unable to find IP address 10.1.0.19 on subnet b1f307c5-2a14-474a-ba45-7583a1626a55 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource http://logs.openstack.org/17/355117/9/check/gate-tempest-dsvm-neutron- full-ubuntu- xenial/d6ff45f/logs/screen-q-svc.txt.gz?level=TRACE#_2016-09-10_01_04_43_452 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1622616/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
