Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Carl Baldwin]

Nice work, Brian!

On Thu, Jan 22, 2015 at 2:57 PM, Brian Haley  wrote:
> On 01/22/2015 02:35 PM, Kevin Benton wrote:
>> Right, there are two bugs here. One is in whatever went wrong with 
>> defer_apply
>> and one is with this exception handling code. I would allow the fix to go in 
>> for
>> the exception handling and then file another bug for the actual underlying
>> defer_apply bug.
>
> What went wrong with defer_apply() was caused by oslo.concurrency - version
> 1.4.1 seems to fix the problem, see https://review.openstack.org/#/c/149400/
> (thanks Ihar!)
>
> Xavier - can you update your oslo.concurrency to that version and verify it
> helps?  It seems to work in my config.
>
> Then the change in the other patchset could be applied, along with a test that
> triggers exceptions so this gets caught.
>
> Thanks,
>
> -Brian
>
>> On Thu, Jan 22, 2015 at 10:32 AM, Brian Haley > > wrote:
>>
>> On 01/22/2015 01:06 PM, Kevin Benton wrote:
>> > There was a bug for this already.
>> > https://bugs.launchpad.net/bugs/1413111
>>
>> Thanks Kevin.  I added more info to it, but don't think the patch 
>> proposed there
>> is correct.  Something in the iptables manager defer_apply() code isn't
>> quite right.
>>
>> -Brian
>>
>>
>> > On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley > 
>> > >> wrote:
>> >
>> > On 01/22/2015 10:17 AM, Carl Baldwin wrote:
>> > > I think this warrants a bug report.  Could you file one with 
>> what you
>> > > know so far?
>> >
>> > Carl,
>> >
>> > Seems as though a recent change introduced a bug.  This is on a 
>> devstack
>> > I just created today, at l3/vpn-agent startup:
>> >
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent 
>> Traceback (most
>> > recent call last):
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
>> > "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent 
>> return
>> > func(*args, **kwargs)
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
>> > "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
>> process_router
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
>> >  self._process_external(ri)
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
>> > "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
>> _process_external
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
>> >  self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
>> UnboundLocalError:
>> > local variable 'existing_floating_ips' referenced before assignment
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
>> > Traceback (most recent call last):
>> >   File 
>> "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py",
>> line
>> > 82, in _spawn_n_impl
>> > func(*args, **kwargs)
>> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1093, 
>> in
>> > _process_router_update
>> > self._process_router_if_compatible(router)
>> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1047, 
>> in
>> > _process_router_if_compatible
>> > self._process_added_router(router)
>> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1056, 
>> in
>> > _process_added_router
>> > self.process_router(ri)
>> >   File "/opt/stack/neutron/neutron/common/utils.py", line 345, in 
>> call
>> > self.logger(e)
>> >   File
>> "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line
>> > 82, in __exit__
>> > six.reraise(self.type_, self.value, self.tb)
>> >   File "/opt/stack/neutron/neutron/common/utils.py", line 342, in 
>> call
>> > return func(*args, **kwargs)
>> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
>> > process_router
>> > self._process_external(ri)
>> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
>> > _process_external
>> > self._update_fip_statuses(ri, existing_floating_ips, 
>> fip_statuses)
>> > UnboundLocalError: local variable 'existing_floating_ips' 
>> referenced
>> before
>> > assignment
>> >
>> > Since that's happening while we're holding the iptables lock I'm 
>> assuming
>> > no rules are being applied.
>> >
>> > I'm looking into it now, will file a bug if there isn't already 
>> one.
>> >
>>

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Xavier León]

On Thu, Jan 22, 2015 at 10:57 PM, Brian Haley  wrote:
> On 01/22/2015 02:35 PM, Kevin Benton wrote:
>> Right, there are two bugs here. One is in whatever went wrong with 
>> defer_apply
>> and one is with this exception handling code. I would allow the fix to go in 
>> for
>> the exception handling and then file another bug for the actual underlying
>> defer_apply bug.
>
> What went wrong with defer_apply() was caused by oslo.concurrency - version
> 1.4.1 seems to fix the problem, see https://review.openstack.org/#/c/149400/
> (thanks Ihar!)
>
> Xavier - can you update your oslo.concurrency to that version and verify it
> helps?  It seems to work in my config.

Yes. Updating to oslo.concurrency 1.4.1 fixed this problem.

Thanks,
Xavi

>
> Then the change in the other patchset could be applied, along with a test that
> triggers exceptions so this gets caught.
>
> Thanks,
>
> -Brian
>
>> On Thu, Jan 22, 2015 at 10:32 AM, Brian Haley > > wrote:
>>
>> On 01/22/2015 01:06 PM, Kevin Benton wrote:
>> > There was a bug for this already.
>> > https://bugs.launchpad.net/bugs/1413111
>>
>> Thanks Kevin.  I added more info to it, but don't think the patch 
>> proposed there
>> is correct.  Something in the iptables manager defer_apply() code isn't
>> quite right.
>>
>> -Brian
>>
>>
>> > On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley > 
>> > >> wrote:
>> >
>> > On 01/22/2015 10:17 AM, Carl Baldwin wrote:
>> > > I think this warrants a bug report.  Could you file one with 
>> what you
>> > > know so far?
>> >
>> > Carl,
>> >
>> > Seems as though a recent change introduced a bug.  This is on a 
>> devstack
>> > I just created today, at l3/vpn-agent startup:
>> >
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent 
>> Traceback (most
>> > recent call last):
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
>> > "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent 
>> return
>> > func(*args, **kwargs)
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
>> > "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
>> process_router
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
>> >  self._process_external(ri)
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
>> > "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
>> _process_external
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
>> >  self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
>> UnboundLocalError:
>> > local variable 'existing_floating_ips' referenced before assignment
>> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
>> > Traceback (most recent call last):
>> >   File 
>> "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py",
>> line
>> > 82, in _spawn_n_impl
>> > func(*args, **kwargs)
>> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1093, 
>> in
>> > _process_router_update
>> > self._process_router_if_compatible(router)
>> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1047, 
>> in
>> > _process_router_if_compatible
>> > self._process_added_router(router)
>> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1056, 
>> in
>> > _process_added_router
>> > self.process_router(ri)
>> >   File "/opt/stack/neutron/neutron/common/utils.py", line 345, in 
>> call
>> > self.logger(e)
>> >   File
>> "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line
>> > 82, in __exit__
>> > six.reraise(self.type_, self.value, self.tb)
>> >   File "/opt/stack/neutron/neutron/common/utils.py", line 342, in 
>> call
>> > return func(*args, **kwargs)
>> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
>> > process_router
>> > self._process_external(ri)
>> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
>> > _process_external
>> > self._update_fip_statuses(ri, existing_floating_ips, 
>> fip_statuses)
>> > UnboundLocalError: local variable 'existing_floating_ips' 
>> referenced
>> before
>> > assignment
>> >
>> > Since that's happening while we're holding the iptables lock I'm 
>> assuming
>> > no rules are being applied.
>> >
>> > I'm looking into it now, w

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Brian Haley]

On 01/22/2015 02:35 PM, Kevin Benton wrote:
> Right, there are two bugs here. One is in whatever went wrong with defer_apply
> and one is with this exception handling code. I would allow the fix to go in 
> for
> the exception handling and then file another bug for the actual underlying
> defer_apply bug.

What went wrong with defer_apply() was caused by oslo.concurrency - version
1.4.1 seems to fix the problem, see https://review.openstack.org/#/c/149400/
(thanks Ihar!)

Xavier - can you update your oslo.concurrency to that version and verify it
helps?  It seems to work in my config.

Then the change in the other patchset could be applied, along with a test that
triggers exceptions so this gets caught.

Thanks,

-Brian

> On Thu, Jan 22, 2015 at 10:32 AM, Brian Haley  > wrote:
> 
> On 01/22/2015 01:06 PM, Kevin Benton wrote:
> > There was a bug for this already.
> > https://bugs.launchpad.net/bugs/1413111
> 
> Thanks Kevin.  I added more info to it, but don't think the patch 
> proposed there
> is correct.  Something in the iptables manager defer_apply() code isn't
> quite right.
> 
> -Brian
> 
> 
> > On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley  
> > >> wrote:
> >
> > On 01/22/2015 10:17 AM, Carl Baldwin wrote:
> > > I think this warrants a bug report.  Could you file one with what 
> you
> > > know so far?
> >
> > Carl,
> >
> > Seems as though a recent change introduced a bug.  This is on a 
> devstack
> > I just created today, at l3/vpn-agent startup:
> >
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback 
> (most
> > recent call last):
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> > "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return
> > func(*args, **kwargs)
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> > "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
> process_router
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> >  self._process_external(ri)
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> > "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
> _process_external
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> >  self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> UnboundLocalError:
> > local variable 'existing_floating_ips' referenced before assignment
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> > Traceback (most recent call last):
> >   File 
> "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py",
> line
> > 82, in _spawn_n_impl
> > func(*args, **kwargs)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1093, in
> > _process_router_update
> > self._process_router_if_compatible(router)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1047, in
> > _process_router_if_compatible
> > self._process_added_router(router)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1056, in
> > _process_added_router
> > self.process_router(ri)
> >   File "/opt/stack/neutron/neutron/common/utils.py", line 345, in 
> call
> > self.logger(e)
> >   File
> "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line
> > 82, in __exit__
> > six.reraise(self.type_, self.value, self.tb)
> >   File "/opt/stack/neutron/neutron/common/utils.py", line 342, in 
> call
> > return func(*args, **kwargs)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
> > process_router
> > self._process_external(ri)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
> > _process_external
> > self._update_fip_statuses(ri, existing_floating_ips, 
> fip_statuses)
> > UnboundLocalError: local variable 'existing_floating_ips' referenced
> before
> > assignment
> >
> > Since that's happening while we're holding the iptables lock I'm 
> assuming
> > no rules are being applied.
> >
> > I'm looking into it now, will file a bug if there isn't already one.
> >
> > -Brian
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.o

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Kevin Benton]

Right, there are two bugs here. One is in whatever went wrong with
defer_apply and one is with this exception handling code. I would allow the
fix to go in for the exception handling and then file another bug for the
actual underlying defer_apply bug.

On Thu, Jan 22, 2015 at 10:32 AM, Brian Haley  wrote:

> On 01/22/2015 01:06 PM, Kevin Benton wrote:
> > There was a bug for this already.
> > https://bugs.launchpad.net/bugs/1413111
>
> Thanks Kevin.  I added more info to it, but don't think the patch proposed
> there
> is correct.  Something in the iptables manager defer_apply() code isn't
> quite right.
>
> -Brian
>
>
> > On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley  > > wrote:
> >
> > On 01/22/2015 10:17 AM, Carl Baldwin wrote:
> > > I think this warrants a bug report.  Could you file one with what
> you
> > > know so far?
> >
> > Carl,
> >
> > Seems as though a recent change introduced a bug.  This is on a
> devstack
> > I just created today, at l3/vpn-agent startup:
> >
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback
> (most
> > recent call last):
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> > "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return
> > func(*args, **kwargs)
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> > "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
> process_router
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> >  self._process_external(ri)
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> > "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
> _process_external
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> >  self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> UnboundLocalError:
> > local variable 'existing_floating_ips' referenced before assignment
> > 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> > Traceback (most recent call last):
> >   File
> "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line
> > 82, in _spawn_n_impl
> > func(*args, **kwargs)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1093, in
> > _process_router_update
> > self._process_router_if_compatible(router)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1047, in
> > _process_router_if_compatible
> > self._process_added_router(router)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1056, in
> > _process_added_router
> > self.process_router(ri)
> >   File "/opt/stack/neutron/neutron/common/utils.py", line 345, in
> call
> > self.logger(e)
> >   File
> "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line
> > 82, in __exit__
> > six.reraise(self.type_, self.value, self.tb)
> >   File "/opt/stack/neutron/neutron/common/utils.py", line 342, in
> call
> > return func(*args, **kwargs)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
> > process_router
> > self._process_external(ri)
> >   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
> > _process_external
> > self._update_fip_statuses(ri, existing_floating_ips,
> fip_statuses)
> > UnboundLocalError: local variable 'existing_floating_ips' referenced
> before
> > assignment
> >
> > Since that's happening while we're holding the iptables lock I'm
> assuming
> > no rules are being applied.
> >
> > I'm looking into it now, will file a bug if there isn't already one.
> >
> > -Brian
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kevin Benton
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Brian Haley]

On 01/22/2015 01:06 PM, Kevin Benton wrote:
> There was a bug for this already.
> https://bugs.launchpad.net/bugs/1413111

Thanks Kevin.  I added more info to it, but don't think the patch proposed there
is correct.  Something in the iptables manager defer_apply() code isn't quite 
right.

-Brian


> On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley  > wrote:
> 
> On 01/22/2015 10:17 AM, Carl Baldwin wrote:
> > I think this warrants a bug report.  Could you file one with what you
> > know so far?
> 
> Carl,
> 
> Seems as though a recent change introduced a bug.  This is on a devstack
> I just created today, at l3/vpn-agent startup:
> 
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most
> recent call last):
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return
> func(*args, **kwargs)
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in 
> process_router
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   
>  self._process_external(ri)
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in 
> _process_external
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   
>  self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent 
> UnboundLocalError:
> local variable 'existing_floating_ips' referenced before assignment
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", 
> line
> 82, in _spawn_n_impl
> func(*args, **kwargs)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1093, in
> _process_router_update
> self._process_router_if_compatible(router)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1047, in
> _process_router_if_compatible
> self._process_added_router(router)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1056, in
> _process_added_router
> self.process_router(ri)
>   File "/opt/stack/neutron/neutron/common/utils.py", line 345, in call
> self.logger(e)
>   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", 
> line
> 82, in __exit__
> six.reraise(self.type_, self.value, self.tb)
>   File "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
> return func(*args, **kwargs)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
> process_router
> self._process_external(ri)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
> _process_external
> self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
> UnboundLocalError: local variable 'existing_floating_ips' referenced 
> before
> assignment
> 
> Since that's happening while we're holding the iptables lock I'm assuming
> no rules are being applied.
> 
> I'm looking into it now, will file a bug if there isn't already one.
> 
> -Brian


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Kevin Benton]

There was a bug for this already.
https://bugs.launchpad.net/bugs/1413111

On Thu, Jan 22, 2015 at 9:07 AM, Brian Haley  wrote:

> On 01/22/2015 10:17 AM, Carl Baldwin wrote:
> > I think this warrants a bug report.  Could you file one with what you
> > know so far?
>
> Carl,
>
> Seems as though a recent change introduced a bug.  This is on a devstack
> I just created today, at l3/vpn-agent startup:
>
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most
> recent call last):
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return
> func(*args, **kwargs)
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in process_router
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
>  self._process_external(ri)
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File
> "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
> _process_external
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
>  self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> UnboundLocalError: local variable 'existing_floating_ips' referenced before
> assignment
> 2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py",
> line 82, in _spawn_n_impl
> func(*args, **kwargs)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1093, in
> _process_router_update
> self._process_router_if_compatible(router)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1047, in
> _process_router_if_compatible
> self._process_added_router(router)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1056, in
> _process_added_router
> self.process_router(ri)
>   File "/opt/stack/neutron/neutron/common/utils.py", line 345, in call
> self.logger(e)
>   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py",
> line 82, in __exit__
> six.reraise(self.type_, self.value, self.tb)
>   File "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
> return func(*args, **kwargs)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in
> process_router
> self._process_external(ri)
>   File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in
> _process_external
> self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
> UnboundLocalError: local variable 'existing_floating_ips' referenced
> before assignment
>
> Since that's happening while we're holding the iptables lock I'm assuming
> no rules are being applied.
>
> I'm looking into it now, will file a bug if there isn't already one.
>
> -Brian
>
>
> > On Wed, Jan 21, 2015 at 2:24 PM, Brian Haley  wrote:
> >> On 01/21/2015 02:29 PM, Xavier León wrote:
> >>> On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley 
> wrote:
>  On 01/20/2015 09:20 AM, Xavier León wrote:
> > Hi all,
> >
> > we've been doing some tests with openstack kilo and found
> > out a problem: iptables routes are not being injected to the
> > router namespace.
> >
> > Scenario:
> > - a private network NOT connected to the outside world.
> > - a router with only one interface connected to the private network.
> > - a vm instance connected to the private network as well.
> >> 
>  Are you sure the l3-agent is running?  You should have seen wrapped
> rules from
>  it in most of these tables, for example:
> 
>  # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015
>  *filter
>  :INPUT ACCEPT [34:10882]
>  :FORWARD ACCEPT [0:0]
>  :OUTPUT ACCEPT [1:84]
>  :neutron-filter-top - [0:0]
>  :neutron-l3-agent-FORWARD - [0:0]
>  :neutron-l3-agent-INPUT - [0:0]
>  :neutron-l3-agent-OUTPUT - [0:0]
>  :neutron-l3-agent-local - [0:0]
>  [...]
> >>>
> >>> Yes, the l3-agent is up and running. I see these rules when executing
> >>> the same test in juno but not in kilo. FYI, it's a all-in-one devstack
> >>> deployment.
> >>>
> 
>  I would check the log files for any errors.
> >>>
> >>> There are no errors in the logs.
> >>>
> >>> After digging a bit more, we have seen that setting the config value
> >>> of enable_isolated_metadata to True (default: False) in dhcp_agent.ini
> >>> solves the problem in our scenario.
> >>> However, this change in configuration was not necessary before (our
> >>> tests passed in juno for that matter with that setting to False). So
> >>> we were wondering if there has been a change in how the metadata
> >>> service is accessed in such scenarios, a new issue because of the l3
> >>> agent refactoring or any other problem in our setup we haven't
> >>> narrowed yet.
> >>
>

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Brian Haley]

On 01/22/2015 10:17 AM, Carl Baldwin wrote:
> I think this warrants a bug report.  Could you file one with what you
> know so far?

Carl,

Seems as though a recent change introduced a bug.  This is on a devstack
I just created today, at l3/vpn-agent startup:

2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent Traceback (most 
recent call last):
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File 
"/opt/stack/neutron/neutron/common/utils.py", line 342, in call
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent return 
func(*args, **kwargs)
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File 
"/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in process_router
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent 
self._process_external(ri)
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent   File 
"/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in _process_external
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent 
self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent UnboundLocalError: 
local variable 'existing_floating_ips' referenced before assignment
2015-01-22 11:55:07.961 4203 TRACE neutron.agent.l3.agent
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, 
in _spawn_n_impl
func(*args, **kwargs)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1093, in 
_process_router_update
self._process_router_if_compatible(router)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1047, in 
_process_router_if_compatible
self._process_added_router(router)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1056, in 
_process_added_router
self.process_router(ri)
  File "/opt/stack/neutron/neutron/common/utils.py", line 345, in call
self.logger(e)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 
82, in __exit__
six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
return func(*args, **kwargs)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 584, in 
process_router
self._process_external(ri)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 576, in 
_process_external
self._update_fip_statuses(ri, existing_floating_ips, fip_statuses)
UnboundLocalError: local variable 'existing_floating_ips' referenced before 
assignment

Since that's happening while we're holding the iptables lock I'm assuming
no rules are being applied.

I'm looking into it now, will file a bug if there isn't already one.

-Brian


> On Wed, Jan 21, 2015 at 2:24 PM, Brian Haley  wrote:
>> On 01/21/2015 02:29 PM, Xavier León wrote:
>>> On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley  wrote:
 On 01/20/2015 09:20 AM, Xavier León wrote:
> Hi all,
>
> we've been doing some tests with openstack kilo and found
> out a problem: iptables routes are not being injected to the
> router namespace.
>
> Scenario:
> - a private network NOT connected to the outside world.
> - a router with only one interface connected to the private network.
> - a vm instance connected to the private network as well.
>> 
 Are you sure the l3-agent is running?  You should have seen wrapped rules 
 from
 it in most of these tables, for example:

 # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015
 *filter
 :INPUT ACCEPT [34:10882]
 :FORWARD ACCEPT [0:0]
 :OUTPUT ACCEPT [1:84]
 :neutron-filter-top - [0:0]
 :neutron-l3-agent-FORWARD - [0:0]
 :neutron-l3-agent-INPUT - [0:0]
 :neutron-l3-agent-OUTPUT - [0:0]
 :neutron-l3-agent-local - [0:0]
 [...]
>>>
>>> Yes, the l3-agent is up and running. I see these rules when executing
>>> the same test in juno but not in kilo. FYI, it's a all-in-one devstack
>>> deployment.
>>>

 I would check the log files for any errors.
>>>
>>> There are no errors in the logs.
>>>
>>> After digging a bit more, we have seen that setting the config value
>>> of enable_isolated_metadata to True (default: False) in dhcp_agent.ini
>>> solves the problem in our scenario.
>>> However, this change in configuration was not necessary before (our
>>> tests passed in juno for that matter with that setting to False). So
>>> we were wondering if there has been a change in how the metadata
>>> service is accessed in such scenarios, a new issue because of the l3
>>> agent refactoring or any other problem in our setup we haven't
>>> narrowed yet.
>>
>> There have been some changes recently in the code, perhaps:
>>
>> https://review.openstack.org/#/c/135467/
>>
>> Or just look at some of the other recent changes in the repository?
>>
>> -Brian


__
OpenStack Development Mailing List (not for usage qu

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Carl Baldwin]

I think this warrants a bug report.  Could you file one with what you
know so far?

Carl

On Wed, Jan 21, 2015 at 2:24 PM, Brian Haley  wrote:
> On 01/21/2015 02:29 PM, Xavier León wrote:
>> On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley  wrote:
>>> On 01/20/2015 09:20 AM, Xavier León wrote:
 Hi all,

 we've been doing some tests with openstack kilo and found
 out a problem: iptables routes are not being injected to the
 router namespace.

 Scenario:
 - a private network NOT connected to the outside world.
 - a router with only one interface connected to the private network.
 - a vm instance connected to the private network as well.
> 
>>> Are you sure the l3-agent is running?  You should have seen wrapped rules 
>>> from
>>> it in most of these tables, for example:
>>>
>>> # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015
>>> *filter
>>> :INPUT ACCEPT [34:10882]
>>> :FORWARD ACCEPT [0:0]
>>> :OUTPUT ACCEPT [1:84]
>>> :neutron-filter-top - [0:0]
>>> :neutron-l3-agent-FORWARD - [0:0]
>>> :neutron-l3-agent-INPUT - [0:0]
>>> :neutron-l3-agent-OUTPUT - [0:0]
>>> :neutron-l3-agent-local - [0:0]
>>> [...]
>>
>> Yes, the l3-agent is up and running. I see these rules when executing
>> the same test in juno but not in kilo. FYI, it's a all-in-one devstack
>> deployment.
>>
>>>
>>> I would check the log files for any errors.
>>
>> There are no errors in the logs.
>>
>> After digging a bit more, we have seen that setting the config value
>> of enable_isolated_metadata to True (default: False) in dhcp_agent.ini
>> solves the problem in our scenario.
>> However, this change in configuration was not necessary before (our
>> tests passed in juno for that matter with that setting to False). So
>> we were wondering if there has been a change in how the metadata
>> service is accessed in such scenarios, a new issue because of the l3
>> agent refactoring or any other problem in our setup we haven't
>> narrowed yet.
>
> There have been some changes recently in the code, perhaps:
>
> https://review.openstack.org/#/c/135467/
>
> Or just look at some of the other recent changes in the repository?
>
> -Brian
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Brian Haley]

On 01/21/2015 02:29 PM, Xavier León wrote:
> On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley  wrote:
>> On 01/20/2015 09:20 AM, Xavier León wrote:
>>> Hi all,
>>>
>>> we've been doing some tests with openstack kilo and found
>>> out a problem: iptables routes are not being injected to the
>>> router namespace.
>>>
>>> Scenario:
>>> - a private network NOT connected to the outside world.
>>> - a router with only one interface connected to the private network.
>>> - a vm instance connected to the private network as well.

>> Are you sure the l3-agent is running?  You should have seen wrapped rules 
>> from
>> it in most of these tables, for example:
>>
>> # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015
>> *filter
>> :INPUT ACCEPT [34:10882]
>> :FORWARD ACCEPT [0:0]
>> :OUTPUT ACCEPT [1:84]
>> :neutron-filter-top - [0:0]
>> :neutron-l3-agent-FORWARD - [0:0]
>> :neutron-l3-agent-INPUT - [0:0]
>> :neutron-l3-agent-OUTPUT - [0:0]
>> :neutron-l3-agent-local - [0:0]
>> [...]
> 
> Yes, the l3-agent is up and running. I see these rules when executing
> the same test in juno but not in kilo. FYI, it's a all-in-one devstack
> deployment.
> 
>>
>> I would check the log files for any errors.
> 
> There are no errors in the logs.
> 
> After digging a bit more, we have seen that setting the config value
> of enable_isolated_metadata to True (default: False) in dhcp_agent.ini
> solves the problem in our scenario.
> However, this change in configuration was not necessary before (our
> tests passed in juno for that matter with that setting to False). So
> we were wondering if there has been a change in how the metadata
> service is accessed in such scenarios, a new issue because of the l3
> agent refactoring or any other problem in our setup we haven't
> narrowed yet.

There have been some changes recently in the code, perhaps:

https://review.openstack.org/#/c/135467/

Or just look at some of the other recent changes in the repository?

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Xavier León]

On Tue, Jan 20, 2015 at 10:32 PM, Brian Haley  wrote:
> On 01/20/2015 09:20 AM, Xavier León wrote:
>> Hi all,
>>
>> we've been doing some tests with openstack kilo and found
>> out a problem: iptables routes are not being injected to the
>> router namespace.
>>
>> Scenario:
>> - a private network NOT connected to the outside world.
>> - a router with only one interface connected to the private network.
>> - a vm instance connected to the private network as well.
>>
>> From inside the instance, we try to get some information from the
>> metadata service with curl:
>>
>> $ curl http://169.254.169.254
>> curl: (7) couldn't connect to host
>>
>> With the same set up in juno, there was no such problem and
>> metadata information is shown.
>>
>> The request is not filtered at the instance and hits the router
>> namespace (checked with tcpdump). However, when looking
>> from the controller at the iptables rules at the router, they appear
>> empty.
>>
>> stack@devstack: ~$ sudo ip netns exec
>> qrouter-d4ec737a-c5fb-4f5b-8bd0-1b5353bbade3 iptables-save
> 
>
>> # Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015
>> *filter
>> :INPUT ACCEPT [5:914]
>> :FORWARD ACCEPT [0:0]
>> :OUTPUT ACCEPT [10:868]
>> COMMIT
>
> Are you sure the l3-agent is running?  You should have seen wrapped rules from
> it in most of these tables, for example:
>
> # Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015
> *filter
> :INPUT ACCEPT [34:10882]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [1:84]
> :neutron-filter-top - [0:0]
> :neutron-l3-agent-FORWARD - [0:0]
> :neutron-l3-agent-INPUT - [0:0]
> :neutron-l3-agent-OUTPUT - [0:0]
> :neutron-l3-agent-local - [0:0]
> [...]

Yes, the l3-agent is up and running. I see these rules when executing
the same test in juno but not in kilo. FYI, it's a all-in-one devstack
deployment.

>
> I would check the log files for any errors.

There are no errors in the logs.

After digging a bit more, we have seen that setting the config value
of enable_isolated_metadata to True (default: False) in dhcp_agent.ini
solves the problem in our scenario.
However, this change in configuration was not necessary before (our
tests passed in juno for that matter with that setting to False). So
we were wondering if there has been a change in how the metadata
service is accessed in such scenarios, a new issue because of the l3
agent refactoring or any other problem in our setup we haven't
narrowed yet.

Cheers,
Xavi

>
> -Brian
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Brian Haley]

On 01/20/2015 09:20 AM, Xavier León wrote:
> Hi all,
> 
> we've been doing some tests with openstack kilo and found
> out a problem: iptables routes are not being injected to the
> router namespace.
> 
> Scenario:
> - a private network NOT connected to the outside world.
> - a router with only one interface connected to the private network.
> - a vm instance connected to the private network as well.
> 
> From inside the instance, we try to get some information from the
> metadata service with curl:
> 
> $ curl http://169.254.169.254
> curl: (7) couldn't connect to host
> 
> With the same set up in juno, there was no such problem and
> metadata information is shown.
> 
> The request is not filtered at the instance and hits the router
> namespace (checked with tcpdump). However, when looking
> from the controller at the iptables rules at the router, they appear
> empty.
> 
> stack@devstack: ~$ sudo ip netns exec
> qrouter-d4ec737a-c5fb-4f5b-8bd0-1b5353bbade3 iptables-save


> # Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015
> *filter
> :INPUT ACCEPT [5:914]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [10:868]
> COMMIT

Are you sure the l3-agent is running?  You should have seen wrapped rules from
it in most of these tables, for example:

# Generated by iptables-save v1.4.21 on Tue Jan 20 16:29:19 2015
*filter
:INPUT ACCEPT [34:10882]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1:84]
:neutron-filter-top - [0:0]
:neutron-l3-agent-FORWARD - [0:0]
:neutron-l3-agent-INPUT - [0:0]
:neutron-l3-agent-OUTPUT - [0:0]
:neutron-l3-agent-local - [0:0]
[...]

I would check the log files for any errors.

-Brian

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] iptables routes are not being injected to router namespace

[Xavier León]

Hi all,

we've been doing some tests with openstack kilo and found
out a problem: iptables routes are not being injected to the
router namespace.

Scenario:
- a private network NOT connected to the outside world.
- a router with only one interface connected to the private network.
- a vm instance connected to the private network as well.

>From inside the instance, we try to get some information from the
metadata service with curl:

$ curl http://169.254.169.254
curl: (7) couldn't connect to host

With the same set up in juno, there was no such problem and
metadata information is shown.

The request is not filtered at the instance and hits the router
namespace (checked with tcpdump). However, when looking
from the controller at the iptables rules at the router, they appear
empty.

stack@devstack: ~$ sudo ip netns exec
qrouter-d4ec737a-c5fb-4f5b-8bd0-1b5353bbade3 iptables-save
# Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015
*raw
:PREROUTING ACCEPT [12:1334]
:OUTPUT ACCEPT [10:868]
COMMIT
# Completed on Tue Jan 20 14:05:48 2015
# Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015
*nat
:PREROUTING ACCEPT [10:913]
:INPUT ACCEPT [3:493]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Tue Jan 20 14:05:48 2015
# Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015
*mangle
:PREROUTING ACCEPT [12:1334]
:INPUT ACCEPT [5:914]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [10:868]
:POSTROUTING ACCEPT [10:868]
COMMIT
# Completed on Tue Jan 20 14:05:48 2015
# Generated by iptables-save v1.4.21 on Tue Jan 20 14:05:48 2015
*filter
:INPUT ACCEPT [5:914]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [10:868]
COMMIT

Is this some problem related to the refactoring of the l3 agent?
Any pointer to what might be the problem here? I can provide
more information on the subject if necessary to reproduce this.
Any input would be appreciated.

Cheers,
Xavi

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev