Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace
Nice work, Brian! On Thu, Jan 22, 2015 at 2:57 PM, Brian Haley brian.ha...@hp.com wrote: On 01/22/2015 02:35 PM, Kevin Benton wrote: Right, there are two bugs here. One is in whatever went wrong with defer_apply and one is with this exception handling code. I would allow the fix to go in for the exception handling and then file another bug for the actual underlying defer_apply bug. What went wrong with defer_apply() was caused by oslo.concurrency - version 1.4.1 seems to fix the problem, see https://review.openstack.org/#/c/149400/ (thanks Ihar!) Xavier - can you update your oslo.concurrency to that version and verify it helps? It seems to work in my config. Then the change in the other patchset could be applied, along with a test that triggers exceptions so this gets caught. Thanks, -Brian On Thu, Jan 22, 2015 at 10:32 AM, Brian Haley brian.ha...@hp.com mailto:brian.ha...@hp.com wrote: On 01/22/2015 01:06 PM, Kevin Benton wrote: There was a bug for this already. https://bugs.launchpad.net/bugs/1413111 Thanks Kevin. I added more info to it, but don't think the patch proposed there is correct. Something in the iptables manager defer_apply() code isn't quite right. -Brian On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley brian.ha...@hp.com mailto:brian.ha...@hp.com mailto:brian.ha...@hp.com mailto:brian.ha...@hp.com wrote: On 01/22/2015 10:17 AM, Carl Baldwin wrote: I think this warrants a bug report. Could you file one with what you know so far? Carl, Seems as though a recent change introduced a bug. This is on a devstack I just created today, at l3/vpn-agent startup: 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/common/utils.py, line 342, in call 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return func(*args, **kwargs) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._process_external(ri) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py, line 82, in _spawn_n_impl func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1093, in _process_router_update self._process_router_if_compatible(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1047, in _process_router_if_compatible self._process_added_router(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1056, in _process_added_router self.process_router(ri) File /opt/stack/neutron/neutron/common/utils.py, line 345, in call self.logger(e) File /usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py, line 82, in __exit__ six.reraise(self.type_, self.value, self.tb) File /opt/stack/neutron/neutron/common/utils.py, line 342, in call return func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router self._process_external(ri) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment Since that's happening while we're holding the iptables lock I'm assuming no rules are being applied. I'm looking into it now, will file a bug if there isn't already one. -Brian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace
On Thu, Jan 22, 2015 at 10:57 PM, Brian Haley brian.ha...@hp.com wrote: On 01/22/2015 02:35 PM, Kevin Benton wrote: Right, there are two bugs here. One is in whatever went wrong with defer_apply and one is with this exception handling code. I would allow the fix to go in for the exception handling and then file another bug for the actual underlying defer_apply bug. What went wrong with defer_apply() was caused by oslo.concurrency - version 1.4.1 seems to fix the problem, see https://review.openstack.org/#/c/149400/ (thanks Ihar!) Xavier - can you update your oslo.concurrency to that version and verify it helps? It seems to work in my config. Yes. Updating to oslo.concurrency 1.4.1 fixed this problem. Thanks, Xavi Then the change in the other patchset could be applied, along with a test that triggers exceptions so this gets caught. Thanks, -Brian On Thu, Jan 22, 2015 at 10:32 AM, Brian Haley brian.ha...@hp.com mailto:brian.ha...@hp.com wrote: On 01/22/2015 01:06 PM, Kevin Benton wrote: There was a bug for this already. https://bugs.launchpad.net/bugs/1413111 Thanks Kevin. I added more info to it, but don't think the patch proposed there is correct. Something in the iptables manager defer_apply() code isn't quite right. -Brian On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley brian.ha...@hp.com mailto:brian.ha...@hp.com mailto:brian.ha...@hp.com mailto:brian.ha...@hp.com wrote: On 01/22/2015 10:17 AM, Carl Baldwin wrote: I think this warrants a bug report. Could you file one with what you know so far? Carl, Seems as though a recent change introduced a bug. This is on a devstack I just created today, at l3/vpn-agent startup: 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/common/utils.py, line 342, in call 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return func(*args, **kwargs) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._process_external(ri) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py, line 82, in _spawn_n_impl func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1093, in _process_router_update self._process_router_if_compatible(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1047, in _process_router_if_compatible self._process_added_router(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1056, in _process_added_router self.process_router(ri) File /opt/stack/neutron/neutron/common/utils.py, line 345, in call self.logger(e) File /usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py, line 82, in __exit__ six.reraise(self.type_, self.value, self.tb) File /opt/stack/neutron/neutron/common/utils.py, line 342, in call return func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router self._process_external(ri) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment Since that's happening while we're holding the iptables lock I'm assuming no rules are being applied. I'm looking into it now, will file a bug if there isn't already one. -Brian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:
Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace
On 01/22/2015 01:06 PM, Kevin Benton wrote: There was a bug for this already. https://bugs.launchpad.net/bugs/1413111 Thanks Kevin. I added more info to it, but don't think the patch proposed there is correct. Something in the iptables manager defer_apply() code isn't quite right. -Brian On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley brian.ha...@hp.com mailto:brian.ha...@hp.com wrote: On 01/22/2015 10:17 AM, Carl Baldwin wrote: I think this warrants a bug report. Could you file one with what you know so far? Carl, Seems as though a recent change introduced a bug. This is on a devstack I just created today, at l3/vpn-agent startup: 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/common/utils.py, line 342, in call 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return func(*args, **kwargs) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._process_external(ri) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py, line 82, in _spawn_n_impl func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1093, in _process_router_update self._process_router_if_compatible(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1047, in _process_router_if_compatible self._process_added_router(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1056, in _process_added_router self.process_router(ri) File /opt/stack/neutron/neutron/common/utils.py, line 345, in call self.logger(e) File /usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py, line 82, in __exit__ six.reraise(self.type_, self.value, self.tb) File /opt/stack/neutron/neutron/common/utils.py, line 342, in call return func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router self._process_external(ri) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment Since that's happening while we're holding the iptables lock I'm assuming no rules are being applied. I'm looking into it now, will file a bug if there isn't already one. -Brian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace
On 01/22/2015 02:35 PM, Kevin Benton wrote: Right, there are two bugs here. One is in whatever went wrong with defer_apply and one is with this exception handling code. I would allow the fix to go in for the exception handling and then file another bug for the actual underlying defer_apply bug. What went wrong with defer_apply() was caused by oslo.concurrency - version 1.4.1 seems to fix the problem, see https://review.openstack.org/#/c/149400/ (thanks Ihar!) Xavier - can you update your oslo.concurrency to that version and verify it helps? It seems to work in my config. Then the change in the other patchset could be applied, along with a test that triggers exceptions so this gets caught. Thanks, -Brian On Thu, Jan 22, 2015 at 10:32 AM, Brian Haley brian.ha...@hp.com mailto:brian.ha...@hp.com wrote: On 01/22/2015 01:06 PM, Kevin Benton wrote: There was a bug for this already. https://bugs.launchpad.net/bugs/1413111 Thanks Kevin. I added more info to it, but don't think the patch proposed there is correct. Something in the iptables manager defer_apply() code isn't quite right. -Brian On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley brian.ha...@hp.com mailto:brian.ha...@hp.com mailto:brian.ha...@hp.com mailto:brian.ha...@hp.com wrote: On 01/22/2015 10:17 AM, Carl Baldwin wrote: I think this warrants a bug report. Could you file one with what you know so far? Carl, Seems as though a recent change introduced a bug. This is on a devstack I just created today, at l3/vpn-agent startup: 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/common/utils.py, line 342, in call 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return func(*args, **kwargs) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._process_external(ri) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py, line 82, in _spawn_n_impl func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1093, in _process_router_update self._process_router_if_compatible(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1047, in _process_router_if_compatible self._process_added_router(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1056, in _process_added_router self.process_router(ri) File /opt/stack/neutron/neutron/common/utils.py, line 345, in call self.logger(e) File /usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py, line 82, in __exit__ six.reraise(self.type_, self.value, self.tb) File /opt/stack/neutron/neutron/common/utils.py, line 342, in call return func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router self._process_external(ri) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment Since that's happening while we're holding the iptables lock I'm assuming no rules are being applied. I'm looking into it now, will file a bug if there isn't already one. -Brian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace
I think this warrants a bug report. Could you file one with what you know so far? Carl On Wed, Jan 21, 2015 at 2:24 PM, Brian Haley brian.ha...@hp.com wrote: On 01/21/2015 02:29 PM, Xavier León wrote: On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley brian.ha...@hp.com wrote: On 01/20/2015 09:20 AM, Xavier León wrote: Hi all, we've been doing some tests with openstack kilo and found out a problem: iptables routes are not being injected to the router namespace. Scenario: - a private network NOT connected to the outside world. - a router with only one interface connected to the private network. - a vm instance connected to the private network as well. snip Are you sure the l3-agent is running? You should have seen wrapped rules from it in most of these tables, for example: # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015 *filter :INPUT ACCEPT [34:10882] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1:84] :neutron-filter-top - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-local - [0:0] [...] Yes, the l3-agent is up and running. I see these rules when executing the same test in juno but not in kilo. FYI, it's a all-in-one devstack deployment. I would check the log files for any errors. There are no errors in the logs. After digging a bit more, we have seen that setting the config value of enable_isolated_metadata to True (default: False) in dhcp_agent.ini solves the problem in our scenario. However, this change in configuration was not necessary before (our tests passed in juno for that matter with that setting to False). So we were wondering if there has been a change in how the metadata service is accessed in such scenarios, a new issue because of the l3 agent refactoring or any other problem in our setup we haven't narrowed yet. There have been some changes recently in the code, perhaps: https://review.openstack.org/#/c/135467/ Or just look at some of the other recent changes in the repository? -Brian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace
Right, there are two bugs here. One is in whatever went wrong with defer_apply and one is with this exception handling code. I would allow the fix to go in for the exception handling and then file another bug for the actual underlying defer_apply bug. On Thu, Jan 22, 2015 at 10:32 AM, Brian Haley brian.ha...@hp.com wrote: On 01/22/2015 01:06 PM, Kevin Benton wrote: There was a bug for this already. https://bugs.launchpad.net/bugs/1413111 Thanks Kevin. I added more info to it, but don't think the patch proposed there is correct. Something in the iptables manager defer_apply() code isn't quite right. -Brian On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley brian.ha...@hp.com mailto:brian.ha...@hp.com wrote: On 01/22/2015 10:17 AM, Carl Baldwin wrote: I think this warrants a bug report. Could you file one with what you know so far? Carl, Seems as though a recent change introduced a bug. This is on a devstack I just created today, at l3/vpn-agent startup: 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/common/utils.py, line 342, in call 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return func(*args, **kwargs) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._process_external(ri) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py, line 82, in _spawn_n_impl func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1093, in _process_router_update self._process_router_if_compatible(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1047, in _process_router_if_compatible self._process_added_router(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1056, in _process_added_router self.process_router(ri) File /opt/stack/neutron/neutron/common/utils.py, line 345, in call self.logger(e) File /usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py, line 82, in __exit__ six.reraise(self.type_, self.value, self.tb) File /opt/stack/neutron/neutron/common/utils.py, line 342, in call return func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router self._process_external(ri) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment Since that's happening while we're holding the iptables lock I'm assuming no rules are being applied. I'm looking into it now, will file a bug if there isn't already one. -Brian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Kevin Benton __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace
On 01/22/2015 10:17 AM, Carl Baldwin wrote: I think this warrants a bug report. Could you file one with what you know so far? Carl, Seems as though a recent change introduced a bug. This is on a devstack I just created today, at l3/vpn-agent startup: 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/common/utils.py, line 342, in call 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return func(*args, **kwargs) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._process_external(ri) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py, line 82, in _spawn_n_impl func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1093, in _process_router_update self._process_router_if_compatible(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1047, in _process_router_if_compatible self._process_added_router(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1056, in _process_added_router self.process_router(ri) File /opt/stack/neutron/neutron/common/utils.py, line 345, in call self.logger(e) File /usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py, line 82, in __exit__ six.reraise(self.type_, self.value, self.tb) File /opt/stack/neutron/neutron/common/utils.py, line 342, in call return func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router self._process_external(ri) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment Since that's happening while we're holding the iptables lock I'm assuming no rules are being applied. I'm looking into it now, will file a bug if there isn't already one. -Brian On Wed, Jan 21, 2015 at 2:24 PM, Brian Haley brian.ha...@hp.com wrote: On 01/21/2015 02:29 PM, Xavier León wrote: On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley brian.ha...@hp.com wrote: On 01/20/2015 09:20 AM, Xavier León wrote: Hi all, we've been doing some tests with openstack kilo and found out a problem: iptables routes are not being injected to the router namespace. Scenario: - a private network NOT connected to the outside world. - a router with only one interface connected to the private network. - a vm instance connected to the private network as well. snip Are you sure the l3-agent is running? You should have seen wrapped rules from it in most of these tables, for example: # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015 *filter :INPUT ACCEPT [34:10882] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1:84] :neutron-filter-top - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-local - [0:0] [...] Yes, the l3-agent is up and running. I see these rules when executing the same test in juno but not in kilo. FYI, it's a all-in-one devstack deployment. I would check the log files for any errors. There are no errors in the logs. After digging a bit more, we have seen that setting the config value of enable_isolated_metadata to True (default: False) in dhcp_agent.ini solves the problem in our scenario. However, this change in configuration was not necessary before (our tests passed in juno for that matter with that setting to False). So we were wondering if there has been a change in how the metadata service is accessed in such scenarios, a new issue because of the l3 agent refactoring or any other problem in our setup we haven't narrowed yet. There have been some changes recently in the code, perhaps: https://review.openstack.org/#/c/135467/ Or just look at some of the other recent changes in the repository? -Brian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace
There was a bug for this already. https://bugs.launchpad.net/bugs/1413111 On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley brian.ha...@hp.com wrote: On 01/22/2015 10:17 AM, Carl Baldwin wrote: I think this warrants a bug report. Could you file one with what you know so far? Carl, Seems as though a recent change introduced a bug. This is on a devstack I just created today, at l3/vpn-agent startup: 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/common/utils.py, line 342, in call 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return func(*args, **kwargs) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._process_external(ri) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most recent call last): File /usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py, line 82, in _spawn_n_impl func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1093, in _process_router_update self._process_router_if_compatible(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1047, in _process_router_if_compatible self._process_added_router(router) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 1056, in _process_added_router self.process_router(ri) File /opt/stack/neutron/neutron/common/utils.py, line 345, in call self.logger(e) File /usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py, line 82, in __exit__ six.reraise(self.type_, self.value, self.tb) File /opt/stack/neutron/neutron/common/utils.py, line 342, in call return func(*args, **kwargs) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 584, in process_router self._process_external(ri) File /opt/stack/neutron/neutron/agent/l3/agent.py, line 576, in _process_external self._update_fip_statuses(ri, existing_floating_ips, fip_statuses) UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment Since that's happening while we're holding the iptables lock I'm assuming no rules are being applied. I'm looking into it now, will file a bug if there isn't already one. -Brian On Wed, Jan 21, 2015 at 2:24 PM, Brian Haley brian.ha...@hp.com wrote: On 01/21/2015 02:29 PM, Xavier León wrote: On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley brian.ha...@hp.com wrote: On 01/20/2015 09:20 AM, Xavier León wrote: Hi all, we've been doing some tests with openstack kilo and found out a problem: iptables routes are not being injected to the router namespace. Scenario: - a private network NOT connected to the outside world. - a router with only one interface connected to the private network. - a vm instance connected to the private network as well. snip Are you sure the l3-agent is running? You should have seen wrapped rules from it in most of these tables, for example: # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015 *filter :INPUT ACCEPT [34:10882] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1:84] :neutron-filter-top - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-local - [0:0] [...] Yes, the l3-agent is up and running. I see these rules when executing the same test in juno but not in kilo. FYI, it's a all-in-one devstack deployment. I would check the log files for any errors. There are no errors in the logs. After digging a bit more, we have seen that setting the config value of enable_isolated_metadata to True (default: False) in dhcp_agent.ini solves the problem in our scenario. However, this change in configuration was not necessary before (our tests passed in juno for that matter with that setting to False). So we were wondering if there has been a change in how the metadata service is accessed in such scenarios, a new issue because of the l3 agent refactoring or any other problem in our setup we haven't narrowed yet. There have been some changes recently in the code, perhaps: https://review.openstack.org/#/c/135467/ Or just look at some of the other recent changes in the repository? -Brian
Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace
On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley brian.ha...@hp.com wrote: On 01/20/2015 09:20 AM, Xavier León wrote: Hi all, we've been doing some tests with openstack kilo and found out a problem: iptables routes are not being injected to the router namespace. Scenario: - a private network NOT connected to the outside world. - a router with only one interface connected to the private network. - a vm instance connected to the private network as well. From inside the instance, we try to get some information from the metadata service with curl: $ curl http://169.254.169.254 curl: (7) couldn't connect to host With the same set up in juno, there was no such problem and metadata information is shown. The request is not filtered at the instance and hits the router namespace (checked with tcpdump). However, when looking from the controller at the iptables rules at the router, they appear empty. stack@devstack: ~$ sudo ip netns exec qrouter-d4ec737a-c5fb-4f5b-8bd0-1b5353bbade3 iptables-save snip # Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015 *filter :INPUT ACCEPT [5:914] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [10:868] COMMIT Are you sure the l3-agent is running? You should have seen wrapped rules from it in most of these tables, for example: # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015 *filter :INPUT ACCEPT [34:10882] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1:84] :neutron-filter-top - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-local - [0:0] [...] Yes, the l3-agent is up and running. I see these rules when executing the same test in juno but not in kilo. FYI, it's a all-in-one devstack deployment. I would check the log files for any errors. There are no errors in the logs. After digging a bit more, we have seen that setting the config value of enable_isolated_metadata to True (default: False) in dhcp_agent.ini solves the problem in our scenario. However, this change in configuration was not necessary before (our tests passed in juno for that matter with that setting to False). So we were wondering if there has been a change in how the metadata service is accessed in such scenarios, a new issue because of the l3 agent refactoring or any other problem in our setup we haven't narrowed yet. Cheers, Xavi -Brian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace
On 01/21/2015 02:29 PM, Xavier León wrote: On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley brian.ha...@hp.com wrote: On 01/20/2015 09:20 AM, Xavier León wrote: Hi all, we've been doing some tests with openstack kilo and found out a problem: iptables routes are not being injected to the router namespace. Scenario: - a private network NOT connected to the outside world. - a router with only one interface connected to the private network. - a vm instance connected to the private network as well. snip Are you sure the l3-agent is running? You should have seen wrapped rules from it in most of these tables, for example: # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015 *filter :INPUT ACCEPT [34:10882] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1:84] :neutron-filter-top - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-local - [0:0] [...] Yes, the l3-agent is up and running. I see these rules when executing the same test in juno but not in kilo. FYI, it's a all-in-one devstack deployment. I would check the log files for any errors. There are no errors in the logs. After digging a bit more, we have seen that setting the config value of enable_isolated_metadata to True (default: False) in dhcp_agent.ini solves the problem in our scenario. However, this change in configuration was not necessary before (our tests passed in juno for that matter with that setting to False). So we were wondering if there has been a change in how the metadata service is accessed in such scenarios, a new issue because of the l3 agent refactoring or any other problem in our setup we haven't narrowed yet. There have been some changes recently in the code, perhaps: https://review.openstack.org/#/c/135467/ Or just look at some of the other recent changes in the repository? -Brian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron] iptables routes are not being injected to router namespace
Hi all, we've been doing some tests with openstack kilo and found out a problem: iptables routes are not being injected to the router namespace. Scenario: - a private network NOT connected to the outside world. - a router with only one interface connected to the private network. - a vm instance connected to the private network as well. From inside the instance, we try to get some information from the metadata service with curl: $ curl http://169.254.169.254 curl: (7) couldn't connect to host With the same set up in juno, there was no such problem and metadata information is shown. The request is not filtered at the instance and hits the router namespace (checked with tcpdump). However, when looking from the controller at the iptables rules at the router, they appear empty. stack@devstack: ~$ sudo ip netns exec qrouter-d4ec737a-c5fb-4f5b-8bd0-1b5353bbade3 iptables-save # Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015 *raw :PREROUTING ACCEPT [12:1334] :OUTPUT ACCEPT [10:868] COMMIT # Completed on Tue Jan 20 14:05:48 2015 # Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015 *nat :PREROUTING ACCEPT [10:913] :INPUT ACCEPT [3:493] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] COMMIT # Completed on Tue Jan 20 14:05:48 2015 # Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015 *mangle :PREROUTING ACCEPT [12:1334] :INPUT ACCEPT [5:914] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [10:868] :POSTROUTING ACCEPT [10:868] COMMIT # Completed on Tue Jan 20 14:05:48 2015 # Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015 *filter :INPUT ACCEPT [5:914] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [10:868] COMMIT Is this some problem related to the refactoring of the l3 agent? Any pointer to what might be the problem here? I can provide more information on the subject if necessary to reproduce this. Any input would be appreciated. Cheers, Xavi __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace
On 01/20/2015 09:20 AM, Xavier León wrote: Hi all, we've been doing some tests with openstack kilo and found out a problem: iptables routes are not being injected to the router namespace. Scenario: - a private network NOT connected to the outside world. - a router with only one interface connected to the private network. - a vm instance connected to the private network as well. From inside the instance, we try to get some information from the metadata service with curl: $ curl http://169.254.169.254 curl: (7) couldn't connect to host With the same set up in juno, there was no such problem and metadata information is shown. The request is not filtered at the instance and hits the router namespace (checked with tcpdump). However, when looking from the controller at the iptables rules at the router, they appear empty. stack@devstack: ~$ sudo ip netns exec qrouter-d4ec737a-c5fb-4f5b-8bd0-1b5353bbade3 iptables-save snip # Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015 *filter :INPUT ACCEPT [5:914] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [10:868] COMMIT Are you sure the l3-agent is running? You should have seen wrapped rules from it in most of these tables, for example: # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015 *filter :INPUT ACCEPT [34:10882] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [1:84] :neutron-filter-top - [0:0] :neutron-l3-agent-FORWARD - [0:0] :neutron-l3-agent-INPUT - [0:0] :neutron-l3-agent-OUTPUT - [0:0] :neutron-l3-agent-local - [0:0] [...] I would check the log files for any errors. -Brian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
