Re: [openstack-dev] [neutron] Neutron L3 agent/Keepalived

[Anil Venkata]

Which neutron branch you are using?
May be this bug https://bugs.launchpad.net/neutron/+bug/1723848 is similar
and
corresponding https://review.openstack.org/#/c/512179/ patch was merged in
u/s
and backported to Ocata and pike branches.

Please also check these [1] relevant patches
[1]
https://review.openstack.org/#/q/topic:bug/1597461+(status:open+OR+status:merged)

Thanks
Anilvenkata

On Mon, Dec 4, 2017 at 6:38 PM, Anna Taraday 
wrote:

> Please, file a bug about that on https://bugs.launchpad.net/neutron/+bugs
> with tag "l3-ha".
>
> On Thu, Nov 30, 2017 at 6:00 AM Ajay Kalambur (akalambu) <
> akala...@cisco.com> wrote:
>
>> I noticed that this happens when the router HA interface shows status:
>> Down what can cause the ha interface to do down
>>
>> Ajay
>>
>>
>> From: Ajay Kalambur 
>> Reply-To: "OpenStack Development Mailing List (not for usage questions)"
>> 
>> Date: Wednesday, November 29, 2017 at 4:14 PM
>> To: "OpenStack Development Mailing List (not for usage questions)" <
>> openstack-dev@lists.openstack.org>
>> Subject: [openstack-dev] [neutron] Neutron L3 agent/Keepalived
>>
>> Hi
>> I have a case where after running the system for a while I see that
>> floating ip association API gets accepted and no Errors in neutron l3 logs
>> But when I go into the qrouter namespace and check the qg- interface the
>> floating ip is not added
>> Also the keepalived.conf is not updated and SIGUP of the keepalived
>> process is not done
>>
>> So whats the place to look in this case.  What is the flow in neutron l3
>> agent who adds the floating ip to the qg- interface
>> I see the router update notification received
>> Key log snippet attached. As you can see SIGUP of keepalived is missing
>> and also I confirmed keepalived configs are not updated
>>
>>
>> 2017-11-29 14:32:08.883 78 DEBUG oslo_messaging._drivers.amqpdriver [-]
>> received message with unique_id: b3886b998c3049e799170bb5351300d1
>> __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_
>> drivers/amqpdriver.py:269
>> 2017-11-29 14:32:08.885 78 DEBUG neutron.agent.l3.agent
>> [req-6c505d32-2d1a-4392-8ea3-0bb888397b9e e759eebecc4648b4964d4ecf439dc0ff
>> 13b4f6fb188048ba9bbf211344e3342f - - -] Got routers updated notification
>> :[u'a1a59e5f-d74e-404d-a3aa-1667c4442aed'] routers_updated
>> /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:413
>> 2017-11-29 14:32:08.886 78 DEBUG neutron.agent.l3.agent [-] Starting
>> router update for a1a59e5f-d74e-404d-a3aa-1667c4442aed, action None,
>> priority 0 _process_router_update /usr/lib/python2.7/site-
>> packages/neutron/agent/l3/agent.py:487
>> 2017-11-29 14:32:08.886 78 DEBUG oslo_messaging._drivers.amqpdriver [-]
>> CALL msg_id: 92eb015c8c84489681b1efbe949fab8b exchange 'neutron' topic
>> 'q-l3-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_
>> drivers/amqpdriver.py:568
>>
>> 2017-11-29 14:32:09.571 78 DEBUG oslo_messaging._drivers.amqpdriver [-]
>> received reply msg_id: 92eb015c8c84489681b1efbe949fab8b __call__
>> /usr/lib/python2.7/site-packages/oslo_messaging/_
>> drivers/amqpdriver.py:416
>> 2017-11-29 14:32:09.571 78 DEBUG neutron.callbacks.manager [-] Notify
>> callbacks [] for router, before_update _notify_loop /usr/lib/python2.7/site-
>> packages/neutron/callbacks/manager.py:142
>> 2017-11-29 14:32:09.572 78 DEBUG neutron.agent.l3.router_info [-] process
>> router updates process /usr/lib/python2.7/site-packages/neutron/agent/l3/
>> router_info.py:1105
>> 2017-11-29 14:32:09.572 78 DEBUG neutron.agent.linux.utils [-] Running
>> command (rootwrap daemon): ['ip', 'netns', 'exec',
>> 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find',
>> '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
>> execute_rootwrap_daemon /usr/lib/python2.7/site-
>> packages/neutron/agent/linux/utils.py:105
>> 2017-11-29 14:32:09.709 78 DEBUG neutron.agent.linux.utils [-] Exit code:
>> 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/
>> utils.py:150
>> 2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock
>> "l3-agent-pd" acquired by "neutron.agent.linux.pd.sync_router" :: waited
>> 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/
>> lockutils.py:270
>> 2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock
>> "l3-agent-pd" released by "neutron.agent.linux.pd.sync_router" :: held
>> 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/
>> lockutils.py:282
>> 2017-11-29 14:32:09.710 78 DEBUG neutron.agent.linux.utils [-] Running
>> command (rootwrap daemon): ['ip', 'netns', 'exec',
>> 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find',
>> '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
>> execute_rootwrap_daemon /usr/lib/python2.7/site-
>> packages/neutron/agent/linux/utils.py:105
>> 2017-11-29 14:32:09.845 78 DEBUG neutron.agent.linux.utils [-] Exit code:
>> 0 execute 

Re: [openstack-dev] [neutron] Neutron L3 agent/Keepalived

[Anna Taraday]

Please, file a bug about that on https://bugs.launchpad.net/neutron/+bugs
with tag "l3-ha".

On Thu, Nov 30, 2017 at 6:00 AM Ajay Kalambur (akalambu) 
wrote:

> I noticed that this happens when the router HA interface shows status:
> Down what can cause the ha interface to do down
>
> Ajay
>
>
> From: Ajay Kalambur 
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Wednesday, November 29, 2017 at 4:14 PM
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: [openstack-dev] [neutron] Neutron L3 agent/Keepalived
>
> Hi
> I have a case where after running the system for a while I see that
> floating ip association API gets accepted and no Errors in neutron l3 logs
> But when I go into the qrouter namespace and check the qg- interface the
> floating ip is not added
> Also the keepalived.conf is not updated and SIGUP of the keepalived
> process is not done
>
> So whats the place to look in this case.  What is the flow in neutron l3
> agent who adds the floating ip to the qg- interface
> I see the router update notification received
> Key log snippet attached. As you can see SIGUP of keepalived is missing
> and also I confirmed keepalived configs are not updated
>
>
> 2017-11-29 14:32:08.883 78 DEBUG oslo_messaging._drivers.amqpdriver [-]
> received message with unique_id: b3886b998c3049e799170bb5351300d1 __call__
> /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:269
> 2017-11-29 14:32:08.885 78 DEBUG neutron.agent.l3.agent
> [req-6c505d32-2d1a-4392-8ea3-0bb888397b9e e759eebecc4648b4964d4ecf439dc0ff
> 13b4f6fb188048ba9bbf211344e3342f - - -] Got routers updated notification
> :[u'a1a59e5f-d74e-404d-a3aa-1667c4442aed'] routers_updated
> /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:413
> 2017-11-29 14:32:08.886 78 DEBUG neutron.agent.l3.agent [-] Starting
> router update for a1a59e5f-d74e-404d-a3aa-1667c4442aed, action None,
> priority 0 _process_router_update
> /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:487
> 2017-11-29 14:32:08.886 78 DEBUG oslo_messaging._drivers.amqpdriver [-]
> CALL msg_id: 92eb015c8c84489681b1efbe949fab8b exchange 'neutron' topic
> 'q-l3-plugin' _send
> /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568
>
> 2017-11-29 14:32:09.571 78 DEBUG oslo_messaging._drivers.amqpdriver [-]
> received reply msg_id: 92eb015c8c84489681b1efbe949fab8b __call__
> /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416
> 2017-11-29 14:32:09.571 78 DEBUG neutron.callbacks.manager [-] Notify
> callbacks [] for router, before_update _notify_loop
> /usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:142
> 2017-11-29 14:32:09.572 78 DEBUG neutron.agent.l3.router_info [-] process
> router updates process
> /usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:1105
> 2017-11-29 14:32:09.572 78 DEBUG neutron.agent.linux.utils [-] Running
> command (rootwrap daemon): ['ip', 'netns', 'exec',
> 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net',
> '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon
> /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
> 2017-11-29 14:32:09.709 78 DEBUG neutron.agent.linux.utils [-] Exit code:
> 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
> 2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock
> "l3-agent-pd" acquired by "neutron.agent.linux.pd.sync_router" :: waited
> 0.000s inner
> /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
> 2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock
> "l3-agent-pd" released by "neutron.agent.linux.pd.sync_router" :: held
> 0.000s inner
> /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
> 2017-11-29 14:32:09.710 78 DEBUG neutron.agent.linux.utils [-] Running
> command (rootwrap daemon): ['ip', 'netns', 'exec',
> 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net',
> '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon
> /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
> 2017-11-29 14:32:09.845 78 DEBUG neutron.agent.linux.utils [-] Exit code:
> 0 execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
> 2017-11-29 14:32:09.846 78 DEBUG oslo_concurrency.lockutils [-] Acquired
> semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock
> /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
> 2017-11-29 14:32:09.846 78 DEBUG neutron.agent.linux.utils [-] Running
> command (rootwrap daemon): ['ip', 'netns', 'exec',
> 'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-save']
> execute_rootwrap_daemon
> /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
> 2017-11-29 14:32:09.983 78 DEBUG 

Re: [openstack-dev] [neutron] Neutron L3 agent/Keepalived

[Ajay Kalambur (akalambu)]

I noticed that this happens when the router HA interface shows status: Down 
what can cause the ha interface to do down

Ajay


From: Ajay Kalambur >
Reply-To: "OpenStack Development Mailing List (not for usage questions)" 
>
Date: Wednesday, November 29, 2017 at 4:14 PM
To: "OpenStack Development Mailing List (not for usage questions)" 
>
Subject: [openstack-dev] [neutron] Neutron L3 agent/Keepalived

Hi
I have a case where after running the system for a while I see that floating ip 
association API gets accepted and no Errors in neutron l3 logs
But when I go into the qrouter namespace and check the qg- interface the 
floating ip is not added
Also the keepalived.conf is not updated and SIGUP of the keepalived process is 
not done

So whats the place to look in this case.  What is the flow in neutron l3 agent 
who adds the floating ip to the qg- interface
I see the router update notification received
Key log snippet attached. As you can see SIGUP of keepalived is missing and 
also I confirmed keepalived configs are not updated


2017-11-29 14:32:08.883 78 DEBUG oslo_messaging._drivers.amqpdriver [-] 
received message with unique_id: b3886b998c3049e799170bb5351300d1 __call__ 
/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:269
2017-11-29 14:32:08.885 78 DEBUG neutron.agent.l3.agent 
[req-6c505d32-2d1a-4392-8ea3-0bb888397b9e e759eebecc4648b4964d4ecf439dc0ff 
13b4f6fb188048ba9bbf211344e3342f - - -] Got routers updated notification 
:[u'a1a59e5f-d74e-404d-a3aa-1667c4442aed'] routers_updated 
/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:413
2017-11-29 14:32:08.886 78 DEBUG neutron.agent.l3.agent [-] Starting router 
update for a1a59e5f-d74e-404d-a3aa-1667c4442aed, action None, priority 0 
_process_router_update 
/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:487
2017-11-29 14:32:08.886 78 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL 
msg_id: 92eb015c8c84489681b1efbe949fab8b exchange 'neutron' topic 'q-l3-plugin' 
_send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568

2017-11-29 14:32:09.571 78 DEBUG oslo_messaging._drivers.amqpdriver [-] 
received reply msg_id: 92eb015c8c84489681b1efbe949fab8b __call__ 
/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416
2017-11-29 14:32:09.571 78 DEBUG neutron.callbacks.manager [-] Notify callbacks 
[] for router, before_update _notify_loop 
/usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:142
2017-11-29 14:32:09.572 78 DEBUG neutron.agent.l3.router_info [-] process 
router updates process 
/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:1105
2017-11-29 14:32:09.572 78 DEBUG neutron.agent.linux.utils [-] Running command 
(rootwrap daemon): ['ip', 'netns', 'exec', 
'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', 
'-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.709 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 
execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock 
"l3-agent-pd" acquired by "neutron.agent.linux.pd.sync_router" :: waited 0.000s 
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2017-11-29 14:32:09.710 78 DEBUG oslo_concurrency.lockutils [-] Lock 
"l3-agent-pd" released by "neutron.agent.linux.pd.sync_router" :: held 0.000s 
inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2017-11-29 14:32:09.710 78 DEBUG neutron.agent.linux.utils [-] Running command 
(rootwrap daemon): ['ip', 'netns', 'exec', 
'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'find', '/sys/class/net', 
'-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.845 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 
execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.846 78 DEBUG oslo_concurrency.lockutils [-] Acquired 
semaphore "iptables-qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed" lock 
/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2017-11-29 14:32:09.846 78 DEBUG neutron.agent.linux.utils [-] Running command 
(rootwrap daemon): ['ip', 'netns', 'exec', 
'qrouter-a1a59e5f-d74e-404d-a3aa-1667c4442aed', 'iptables-save'] 
execute_rootwrap_daemon 
/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:105
2017-11-29 14:32:09.983 78 DEBUG neutron.agent.linux.utils [-] Exit code: 0 
execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:150
2017-11-29 14:32:09.986 78 DEBUG neutron.agent.linux.utils [-] Running