@Carl, Still happening on rally runs with lots of concurrent subnet deletions in the same network:
http://logs.openstack.org/17/369417/5/check/gate-rally-dsvm-neutron- rally/b0d5b03/logs/screen-q-svc.txt.gz#_2016-09-14_10_58_28_812 ** Changed in: neutron Status: Fix Released => New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1622616 Title: delete_subnet update_port appears racey with ipam Status in neutron: New Bug description: failure spotted in a patch on a delete_subnet call: 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource [req-746d769c-2388-48e0-8e09-38e4190e5364 tempest-PortsTestJSON-432635984 -] delete failed: Exception deleting fixed_ip from port 862b5dea-dca2-4669-b280-867175f5f351 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource Traceback (most recent call last): 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 79, in resource 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource result = method(request=request, **args) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 526, in delete 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return self._delete(request, id, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 87, in wrapped 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise() 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 83, in wrapped 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(*args, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 151, in wrapper 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise() 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 139, in wrapper 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(*args, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 123, in wrapped 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource traceback.format_exc()) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise() 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/api.py", line 118, in wrapped 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 548, in _delete 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource obj_deleter(request.context, id, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(self, context, *args, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1159, in delete_subnet 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource "port %s"), port_id) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise() 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1151, in delete_subnet 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.update_port(context, port_id, data) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/common/utils.py", line 618, in inner 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource return f(self, context, *args, **kwargs) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1370, in update_port 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource port) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 1183, in update_port 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource new_port=new_port) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_backend_mixin.py", line 724, in update_port 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource new_port.get('mac_address')) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 325, in update_port_with_ips 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource new_mac) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 297, in _update_ips_for_port 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource changes.remove) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 93, in _ipam_deallocate_ips 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource "external system for %s"), addresses) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource self.force_reraise() 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/ipam_pluggable_backend.py", line 70, in _ipam_deallocate_ips 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource ipam_subnet.deallocate(ip['ip_address']) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 222, in deallocate 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource ip_address=address) 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource IpAddressAllocationNotFound: Unable to find IP address 10.1.0.19 on subnet b1f307c5-2a14-474a-ba45-7583a1626a55 2016-09-10 01:04:43.452 13725 ERROR neutron.api.v2.resource http://logs.openstack.org/17/355117/9/check/gate-tempest-dsvm-neutron- full-ubuntu- xenial/d6ff45f/logs/screen-q-svc.txt.gz?level=TRACE#_2016-09-10_01_04_43_452 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1622616/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
