Public bug reported: Using Queens in Ubuntu 18.04, with Routed Provider Networks (network segments).
DHCP agent RPC Callback in Neutron Server/API is returning DHCP reserved ports binded to fixed_ips of subnets in other segments, raising the error copied below when DHCP agent starts. In our understanding, this situation arises when DHCP agents are stopped and its network namespace is removed, but it corresponding port not deleted. In our case we installed too many DHCP agents in each segment, so we decided to stop and uninstall several of them, cleaning the Linux network namespaces but not deleting the orphaned dhcp ports. When we tried to reinstall an agent in other compute, we hit the error. --------------- Currently the list of subnets is correctly filtered (https://review.opendev.org/#/c/339365/) and works ok for newly started DHCP agents: subnets = [subnet for subnet in subnets if subnet['network_id'] not in routed_net_ids or subnet['id'] in seg_subnet_ids] ...but the list of ports is not filtered accordingly. We worked around this issue filtering ports immediatly below subnet filter: ports = [ port for port in ports if port['fixed_ips'][0]['subnet_id'] in seg_subnet_ids ] ------------------ ERROR oslo_messaging.rpc.server [req-bd6e9240-7844-4e0b-9c13-ee05e24f01a9 - - - - -] Exception during message handling: HostNotCompatibleWithFixedIps: Host r11-hostxx is not connected to a segment where the existing fixed_ips on port 1aaa7da6-e11f-44d3-adc8-5577359edc96 will function given the routed network topology. ERROR oslo_messaging.rpc.server Traceback (most recent call last): ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner ERROR oslo_messaging.rpc.server return func(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 87, in wrapped ERROR oslo_messaging.rpc.server return f(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 135, in wrapper ERROR oslo_messaging.rpc.server return f(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 122, in wrapped ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 305, in update_dhcp_port ERROR oslo_messaging.rpc.server return self._port_action(plugin, context, port, 'update_port') ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 100, in _port_action ERROR oslo_messaging.rpc.server return plugin.update_port(context, port['id'], port) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 627, in inner ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 161, in wrapped ERROR oslo_messaging.rpc.server return method(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 87, in wrapped ERROR oslo_messaging.rpc.server return f(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 135, in wrapper ERROR oslo_messaging.rpc.server return f(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 122, in wrapped ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 1359, in update_port ERROR oslo_messaging.rpc.server port) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 161, in wrapped ERROR oslo_messaging.rpc.server return method(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1345, in update_port ERROR oslo_messaging.rpc.server new_port=new_port) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/ipam_backend_mixin.py", line 783, in update_port ERROR oslo_messaging.rpc.server host=host, port_id=old_port['id']) ERROR oslo_messaging.rpc.server HostNotCompatibleWithFixedIps: Host r11-hostxx is not connected to a segment where the existing fixed_ips on port 1aaa7da6-e11f-44d3-adc8-5577359edc96 will function given the routed network topology. ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1849726 Title: Routed Networks DHCP agents getting ports in unconnected segments Status in neutron: New Bug description: Using Queens in Ubuntu 18.04, with Routed Provider Networks (network segments). DHCP agent RPC Callback in Neutron Server/API is returning DHCP reserved ports binded to fixed_ips of subnets in other segments, raising the error copied below when DHCP agent starts. In our understanding, this situation arises when DHCP agents are stopped and its network namespace is removed, but it corresponding port not deleted. In our case we installed too many DHCP agents in each segment, so we decided to stop and uninstall several of them, cleaning the Linux network namespaces but not deleting the orphaned dhcp ports. When we tried to reinstall an agent in other compute, we hit the error. --------------- Currently the list of subnets is correctly filtered (https://review.opendev.org/#/c/339365/) and works ok for newly started DHCP agents: subnets = [subnet for subnet in subnets if subnet['network_id'] not in routed_net_ids or subnet['id'] in seg_subnet_ids] ...but the list of ports is not filtered accordingly. We worked around this issue filtering ports immediatly below subnet filter: ports = [ port for port in ports if port['fixed_ips'][0]['subnet_id'] in seg_subnet_ids ] ------------------ ERROR oslo_messaging.rpc.server [req-bd6e9240-7844-4e0b-9c13-ee05e24f01a9 - - - - -] Exception during message handling: HostNotCompatibleWithFixedIps: Host r11-hostxx is not connected to a segment where the existing fixed_ips on port 1aaa7da6-e11f-44d3-adc8-5577359edc96 will function given the routed network topology. ERROR oslo_messaging.rpc.server Traceback (most recent call last): ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner ERROR oslo_messaging.rpc.server return func(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 87, in wrapped ERROR oslo_messaging.rpc.server return f(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 135, in wrapper ERROR oslo_messaging.rpc.server return f(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 122, in wrapped ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 305, in update_dhcp_port ERROR oslo_messaging.rpc.server return self._port_action(plugin, context, port, 'update_port') ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 100, in _port_action ERROR oslo_messaging.rpc.server return plugin.update_port(context, port['id'], port) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 627, in inner ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 161, in wrapped ERROR oslo_messaging.rpc.server return method(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 87, in wrapped ERROR oslo_messaging.rpc.server return f(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 135, in wrapper ERROR oslo_messaging.rpc.server return f(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR oslo_messaging.rpc.server self.force_reraise() ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 122, in wrapped ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 1359, in update_port ERROR oslo_messaging.rpc.server port) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 161, in wrapped ERROR oslo_messaging.rpc.server return method(*args, **kwargs) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1345, in update_port ERROR oslo_messaging.rpc.server new_port=new_port) ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/ipam_backend_mixin.py", line 783, in update_port ERROR oslo_messaging.rpc.server host=host, port_id=old_port['id']) ERROR oslo_messaging.rpc.server HostNotCompatibleWithFixedIps: Host r11-hostxx is not connected to a segment where the existing fixed_ips on port 1aaa7da6-e11f-44d3-adc8-5577359edc96 will function given the routed network topology. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1849726/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
