Re: [openstack-dev] [Neutron][L3][Devstack] Bug during delete floating IPs?
We don't have asserts anywhere else. I don't think it makes sense to only put a couple in. On Sun, Jan 11, 2015 at 8:29 AM, Sunil Kumar wrote: > Actually, I just noticed that a patch which has been pending review > since Sep is not merged, and that actually fixes the root cause. > > https://review.openstack.org/#/c/119269/ > > Having said that, I think the code can definitely use some touching. If > router_ids is not supposed to be None, it should be asserted (enforcing a > contract with the plugin APIs!) instead of letting python crib about it. > Also, the code in plugins/ml2/plugin.py should not even be call > l3plugin.notify_routers_updated if the router_ids set is empty. I think > assert also belongs there. > > -Sunil > -- > *From:* Salvatore Orlando [sorla...@nicira.com] > *Sent:* Sunday, January 11, 2015 7:25 AM > *To:* OpenStack Development Mailing List (not for usage questions) > *Subject:* Re: [openstack-dev] [Neutron][L3][Devstack] Bug during delete > floating IPs? > > I think Kevin is right - the actual root cause lies in plugins for > which disassociate_floating_ips returns None rather than an (empty) > iterable. > > A quick check revealed that at least: > neutron.plugins.embrane.base_plugin.EmbranePlugin > > neutron.plugions.cisco.service_plugins.cisco_router_plugin.CiscoRouterPlugin > > can return None in some circumstances. > I reckon the plugins should be fixed. > While Sunil's patch fixes the issue it does not make a lot of sense since > it call notify_routers_updated with None for routers_id. > I agree with his intention of making the ml2 plugin more robust, but in my > opinion when router_ids is None or empty the notification method should > probably not be called at all. > > Salvatore > > On 11 January 2015 at 12:02, Kevin Benton wrote: > >> That only seems to bury the root cause because router_ids shouldn't be >> none in this case with the reference L3 plugin. >> >> The ML2 plugin calls disassociate_floatingips with do_notify set to >> False so it should always be at least an empty set coming back. >> >> Which L3 plugin were you using? Perhaps the implementation of >> disassociate_floatingips was incorrect. >> >> On Sat, Jan 10, 2015 at 11:42 PM, Sunil Kumar wrote: >> >>> This trivial patch fixes the tracebacks: >>> >>> $ cat disassociate_floating_ips.patch >>> --- neutron/db/l3_db.py.orig2015-01-10 22:20:30.101506298 -0800 >>> +++ neutron/db/l3_db.py 2015-01-10 22:24:18.111479818 -0800 >>> @@ -1257,4 +1257,4 @@ >>> >>> def notify_routers_updated(self, context, router_ids): >>> super(L3_NAT_db_mixin, self).notify_routers_updated( >>> -context, list(router_ids), 'disassociate_floatingips', {}) >>> +context, list(router_ids) if router_ids else None, >>> 'disassociate_floatingips', {}) >>> >>> >>> -Sunil >>> -- >>> *From:* Sunil Kumar [su...@embrane.com] >>> *Sent:* Saturday, January 10, 2015 7:07 PM >>> *To:* OpenStack Development Mailing List (not for usage questions) >>> *Subject:* [openstack-dev] [Neutron][L3][Devstack] Bug during delete >>> floating IPs? >>> >>>Not sure if its something seen by others. I hit this when I run >>> tempest.scenario.test_network_basic_ops.TestNetworkBasicOps against master: >>> >>> 2015-01-10 17:45:13.227 5350 DEBUG neutron.plugins.ml2.plugin >>> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Deleting port >>> e5deb014-0063-4d55-8ee3-5ba3524fee14 delete_port >>> /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:995 >>> 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils >>> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Created new semaphore >>> "db-access" internal_lock >>> /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:206 >>> 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils >>> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Acquired semaphore "db-access" >>> lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:229 >>> 2015-01-10 17:45:13.252 5350 DEBUG neutron.plugins.ml2.plugin >>> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Calling delete_port for >>> e5deb014-0063-4d55-8ee3-5ba3524fee14 owned by network:floatingip >>> delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:1043 >>> 2015-01-10 17:45:13.254 5350 DEBUG neutron.openstack.commo
Re: [openstack-dev] [Neutron][L3][Devstack] Bug during delete floating IPs?
Actually, I just noticed that a patch which has been pending review since Sep is not merged, and that actually fixes the root cause. https://review.openstack.org/#/c/119269/ Having said that, I think the code can definitely use some touching. If router_ids is not supposed to be None, it should be asserted (enforcing a contract with the plugin APIs!) instead of letting python crib about it. Also, the code in plugins/ml2/plugin.py should not even be call l3plugin.notify_routers_updated if the router_ids set is empty. I think assert also belongs there. -Sunil From: Salvatore Orlando [sorla...@nicira.com] Sent: Sunday, January 11, 2015 7:25 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Neutron][L3][Devstack] Bug during delete floating IPs? I think Kevin is right - the actual root cause lies in plugins for which disassociate_floating_ips returns None rather than an (empty) iterable. A quick check revealed that at least: neutron.plugins.embrane.base_plugin.EmbranePlugin neutron.plugions.cisco.service_plugins.cisco_router_plugin.CiscoRouterPlugin can return None in some circumstances. I reckon the plugins should be fixed. While Sunil's patch fixes the issue it does not make a lot of sense since it call notify_routers_updated with None for routers_id. I agree with his intention of making the ml2 plugin more robust, but in my opinion when router_ids is None or empty the notification method should probably not be called at all. Salvatore On 11 January 2015 at 12:02, Kevin Benton mailto:blak...@gmail.com>> wrote: That only seems to bury the root cause because router_ids shouldn't be none in this case with the reference L3 plugin. The ML2 plugin calls disassociate_floatingips with do_notify set to False so it should always be at least an empty set coming back. Which L3 plugin were you using? Perhaps the implementation of disassociate_floatingips was incorrect. On Sat, Jan 10, 2015 at 11:42 PM, Sunil Kumar mailto:su...@embrane.com>> wrote: This trivial patch fixes the tracebacks: $ cat disassociate_floating_ips.patch --- neutron/db/l3_db.py.orig2015-01-10 22:20:30.101506298 -0800 +++ neutron/db/l3_db.py 2015-01-10 22:24:18.111479818 -0800 @@ -1257,4 +1257,4 @@ def notify_routers_updated(self, context, router_ids): super(L3_NAT_db_mixin, self).notify_routers_updated( -context, list(router_ids), 'disassociate_floatingips', {}) +context, list(router_ids) if router_ids else None, 'disassociate_floatingips', {}) -Sunil From: Sunil Kumar [su...@embrane.com<mailto:su...@embrane.com>] Sent: Saturday, January 10, 2015 7:07 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Neutron][L3][Devstack] Bug during delete floating IPs? Not sure if its something seen by others. I hit this when I run tempest.scenario.test_network_basic_ops.TestNetworkBasicOps against master: 2015-01-10 17:45:13.227 5350 DEBUG neutron.plugins.ml2.plugin [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Deleting port e5deb014-0063-4d55-8ee3-5ba3524fee14 delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:995 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Created new semaphore "db-access" internal_lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:206 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Acquired semaphore "db-access" lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:229 2015-01-10 17:45:13.252 5350 DEBUG neutron.plugins.ml2.plugin [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Calling delete_port for e5deb014-0063-4d55-8ee3-5ba3524fee14 owned by network:floatingip delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:1043 2015-01-10 17:45:13.254 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Releasing semaphore "db-access" lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:238 2015-01-10 17:45:13.282 5350 ERROR neutron.api.v2.resource [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] delete failed 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource Traceback (most recent call last): 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource result = method(request=request, **args) 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 479, in delete 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource obj_deleter(request.context, id, **
Re: [openstack-dev] [Neutron][L3][Devstack] Bug during delete floating IPs?
I think Kevin is right - the actual root cause lies in plugins for which disassociate_floating_ips returns None rather than an (empty) iterable. A quick check revealed that at least: neutron.plugins.embrane.base_plugin.EmbranePlugin neutron.plugions.cisco.service_plugins.cisco_router_plugin.CiscoRouterPlugin can return None in some circumstances. I reckon the plugins should be fixed. While Sunil's patch fixes the issue it does not make a lot of sense since it call notify_routers_updated with None for routers_id. I agree with his intention of making the ml2 plugin more robust, but in my opinion when router_ids is None or empty the notification method should probably not be called at all. Salvatore On 11 January 2015 at 12:02, Kevin Benton wrote: > That only seems to bury the root cause because router_ids shouldn't be > none in this case with the reference L3 plugin. > > The ML2 plugin calls disassociate_floatingips with do_notify set to False > so it should always be at least an empty set coming back. > > Which L3 plugin were you using? Perhaps the implementation of > disassociate_floatingips was incorrect. > > On Sat, Jan 10, 2015 at 11:42 PM, Sunil Kumar wrote: > >> This trivial patch fixes the tracebacks: >> >> $ cat disassociate_floating_ips.patch >> --- neutron/db/l3_db.py.orig2015-01-10 22:20:30.101506298 -0800 >> +++ neutron/db/l3_db.py 2015-01-10 22:24:18.111479818 -0800 >> @@ -1257,4 +1257,4 @@ >> >> def notify_routers_updated(self, context, router_ids): >> super(L3_NAT_db_mixin, self).notify_routers_updated( >> -context, list(router_ids), 'disassociate_floatingips', {}) >> +context, list(router_ids) if router_ids else None, >> 'disassociate_floatingips', {}) >> >> >> -Sunil >> -- >> *From:* Sunil Kumar [su...@embrane.com] >> *Sent:* Saturday, January 10, 2015 7:07 PM >> *To:* OpenStack Development Mailing List (not for usage questions) >> *Subject:* [openstack-dev] [Neutron][L3][Devstack] Bug during delete >> floating IPs? >> >> Not sure if its something seen by others. I hit this when I run >> tempest.scenario.test_network_basic_ops.TestNetworkBasicOps against master: >> >> 2015-01-10 17:45:13.227 5350 DEBUG neutron.plugins.ml2.plugin >> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Deleting port >> e5deb014-0063-4d55-8ee3-5ba3524fee14 delete_port >> /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:995 >> 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils >> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Created new semaphore >> "db-access" internal_lock >> /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:206 >> 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils >> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Acquired semaphore "db-access" >> lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:229 >> 2015-01-10 17:45:13.252 5350 DEBUG neutron.plugins.ml2.plugin >> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Calling delete_port for >> e5deb014-0063-4d55-8ee3-5ba3524fee14 owned by network:floatingip >> delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:1043 >> 2015-01-10 17:45:13.254 5350 DEBUG neutron.openstack.common.lockutils >> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Releasing semaphore "db-access" >> lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:238 >> 2015-01-10 17:45:13.282 5350 ERROR neutron.api.v2.resource >> [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] delete failed >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource Traceback >> (most recent call last): >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File >> "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource result = >> method(request=request, **args) >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File >> "/opt/stack/new/neutron/neutron/api/v2/base.py", line 479, in delete >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource >> obj_deleter(request.context, id, **kwargs) >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File >> "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 198, in >> delete_floatingip >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource >> self).delete_floatingip(context, id) >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File >> "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1237, in >> delete_floatingip >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource router_id >> = self._delete_floatingip(context, id) >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File >> "/opt/stack/new/neutron/neutron/db/l3_db.py", line 902, in >> _delete_floatingip >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource >> l3_port_check=False) >> 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File >> "/opt/stack/ne
Re: [openstack-dev] [Neutron][L3][Devstack] Bug during delete floating IPs?
That only seems to bury the root cause because router_ids shouldn't be none in this case with the reference L3 plugin. The ML2 plugin calls disassociate_floatingips with do_notify set to False so it should always be at least an empty set coming back. Which L3 plugin were you using? Perhaps the implementation of disassociate_floatingips was incorrect. On Sat, Jan 10, 2015 at 11:42 PM, Sunil Kumar wrote: > This trivial patch fixes the tracebacks: > > $ cat disassociate_floating_ips.patch > --- neutron/db/l3_db.py.orig2015-01-10 22:20:30.101506298 -0800 > +++ neutron/db/l3_db.py 2015-01-10 22:24:18.111479818 -0800 > @@ -1257,4 +1257,4 @@ > > def notify_routers_updated(self, context, router_ids): > super(L3_NAT_db_mixin, self).notify_routers_updated( > -context, list(router_ids), 'disassociate_floatingips', {}) > +context, list(router_ids) if router_ids else None, > 'disassociate_floatingips', {}) > > > -Sunil > -- > *From:* Sunil Kumar [su...@embrane.com] > *Sent:* Saturday, January 10, 2015 7:07 PM > *To:* OpenStack Development Mailing List (not for usage questions) > *Subject:* [openstack-dev] [Neutron][L3][Devstack] Bug during delete > floating IPs? > > Not sure if its something seen by others. I hit this when I run > tempest.scenario.test_network_basic_ops.TestNetworkBasicOps against master: > > 2015-01-10 17:45:13.227 5350 DEBUG neutron.plugins.ml2.plugin > [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Deleting port > e5deb014-0063-4d55-8ee3-5ba3524fee14 delete_port > /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:995 > 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils > [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Created new semaphore > "db-access" internal_lock > /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:206 > 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils > [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Acquired semaphore "db-access" > lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:229 > 2015-01-10 17:45:13.252 5350 DEBUG neutron.plugins.ml2.plugin > [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Calling delete_port for > e5deb014-0063-4d55-8ee3-5ba3524fee14 owned by network:floatingip > delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:1043 > 2015-01-10 17:45:13.254 5350 DEBUG neutron.openstack.common.lockutils > [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Releasing semaphore "db-access" > lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:238 > 2015-01-10 17:45:13.282 5350 ERROR neutron.api.v2.resource > [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] delete failed > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource Traceback (most > recent call last): > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File > "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource result = > method(request=request, **args) > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File > "/opt/stack/new/neutron/neutron/api/v2/base.py", line 479, in delete > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource > obj_deleter(request.context, id, **kwargs) > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File > "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 198, in > delete_floatingip > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource > self).delete_floatingip(context, id) > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File > "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1237, in > delete_floatingip > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource router_id = > self._delete_floatingip(context, id) > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File > "/opt/stack/new/neutron/neutron/db/l3_db.py", line 902, in > _delete_floatingip > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource > l3_port_check=False) > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File > "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1050, in > delete_port > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource > l3plugin.notify_routers_updated(context, router_ids) > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File > "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1260, in > notify_routers_updated > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource context, > list(router_ids), 'disassociate_floatingips', {}) > 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource TypeError: > 'NoneType' object is not iterable > > Looks like the code is assuming that router_ids can never be None, which > clearly is the case here. Is that a bug? > > Looking elsewhere in the l3_db.py, > L3RpcNotifierMixin.notify_routers_updated() does make a check for > router_ids (which means that that function does
Re: [openstack-dev] [Neutron][L3][Devstack] Bug during delete floating IPs?
This trivial patch fixes the tracebacks: $ cat disassociate_floating_ips.patch --- neutron/db/l3_db.py.orig2015-01-10 22:20:30.101506298 -0800 +++ neutron/db/l3_db.py 2015-01-10 22:24:18.111479818 -0800 @@ -1257,4 +1257,4 @@ def notify_routers_updated(self, context, router_ids): super(L3_NAT_db_mixin, self).notify_routers_updated( -context, list(router_ids), 'disassociate_floatingips', {}) +context, list(router_ids) if router_ids else None, 'disassociate_floatingips', {}) -Sunil From: Sunil Kumar [su...@embrane.com] Sent: Saturday, January 10, 2015 7:07 PM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Neutron][L3][Devstack] Bug during delete floating IPs? Not sure if its something seen by others. I hit this when I run tempest.scenario.test_network_basic_ops.TestNetworkBasicOps against master: 2015-01-10 17:45:13.227 5350 DEBUG neutron.plugins.ml2.plugin [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Deleting port e5deb014-0063-4d55-8ee3-5ba3524fee14 delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:995 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Created new semaphore "db-access" internal_lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:206 2015-01-10 17:45:13.228 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Acquired semaphore "db-access" lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:229 2015-01-10 17:45:13.252 5350 DEBUG neutron.plugins.ml2.plugin [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] Calling delete_port for e5deb014-0063-4d55-8ee3-5ba3524fee14 owned by network:floatingip delete_port /opt/stack/new/neutron/neutron/plugins/ml2/plugin.py:1043 2015-01-10 17:45:13.254 5350 DEBUG neutron.openstack.common.lockutils [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f ] Releasing semaphore "db-access" lock /opt/stack/new/neutron/neutron/openstack/common/lockutils.py:238 2015-01-10 17:45:13.282 5350 ERROR neutron.api.v2.resource [req-2ab4b380-cf3a-4663-90c3-a05ef5f4da0f None] delete failed 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource Traceback (most recent call last): 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/resource.py", line 83, in resource 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource result = method(request=request, **args) 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/api/v2/base.py", line 479, in delete 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource obj_deleter(request.context, id, **kwargs) 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 198, in delete_floatingip 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource self).delete_floatingip(context, id) 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1237, in delete_floatingip 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource router_id = self._delete_floatingip(context, id) 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 902, in _delete_floatingip 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource l3_port_check=False) 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1050, in delete_port 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource l3plugin.notify_routers_updated(context, router_ids) 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1260, in notify_routers_updated 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource context, list(router_ids), 'disassociate_floatingips', {}) 2015-01-10 17:45:13.282 5350 TRACE neutron.api.v2.resource TypeError: 'NoneType' object is not iterable Looks like the code is assuming that router_ids can never be None, which clearly is the case here. Is that a bug? Looking elsewhere in the l3_db.py, L3RpcNotifierMixin.notify_routers_updated() does make a check for router_ids (which means that that function does expect it to be empty some times), but the list() is killing it before it reaches that. This backtrace repeats itself many many times in the neutron logs. Thanks for your help. -Sunil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev