[Yahoo-eng-team] [Bug 1566291] [NEW] L3 agent: at some point an agent becomes unable to handle new routers

Oleg Bondarev Tue, 05 Apr 2016 05:51:08 -0700

Public bug reported:

Following seen in l3 agent logs:


2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent [-] Failed to 
process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent Traceback (most 
recent call last):
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 497, in 
_process_router_update
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     
self._process_router_if_compatible(router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 434, in 
_process_router_if_compatible
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     
self._process_added_router(router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 439, in 
_process_added_router
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     
self._router_added(router['id'], router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 340, in 
_router_added
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     ri = 
self._create_router(router_id, router)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 337, in 
_create_router
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     return 
legacy_router.LegacyRouter(*args, **kwargs)
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 61, in 
__init__
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     
DEFAULT_ADDRESS_SCOPE: ADDRESS_SCOPE_MARK_IDS.pop()}
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent KeyError: 'pop from 
an empty set'
2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
2016-04-05 09:30:09.034 24216 DEBUG neutron.agent.l3.agent [-] Starting router 
update for e341e0e2-5089-46e9-91f9-2099a156b27f, action None, priority 1 
_process_router_update 
/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:463
2016-04-05 09:30:09.035 24216 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL 
msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 exchange 'neutron' topic 'q-l3-plugin' 
_send /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:454
2016-04-05 09:30:09.417 24216 DEBUG oslo_messaging._drivers.amqpdriver [-] 
received reply msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 __call__ 
/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:302
2016-04-05 09:30:09.418 24216 ERROR neutron.agent.l3.agent [-] Failed to 
process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'

So agent is constantly resyncing (causing load on neutron server) and
unable to handle new routers.

I believe that set "ADDRESS_SCOPE_MARK_IDS = set(range(1024, 2048))"
from router_info.py should not be agent global but it should be
ADDRESS_SCOPE_MARK_IDS  per router. Or at least need to return values
back to the set when router is deleted.

** Affects: neutron
     Importance: Undecided
     Assignee: Oleg Bondarev (obondarev)
         Status: New


** Tags: l3-ipam-dhcp

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1566291

Title:
  L3 agent: at some point an agent becomes unable to handle new routers

Status in neutron:
  New

Bug description:
  Following seen in l3 agent logs:

  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent [-] Failed to 
process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent Traceback (most 
recent call last):
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 497, in 
_process_router_update
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     
self._process_router_if_compatible(router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 434, in 
_process_router_if_compatible
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     
self._process_added_router(router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 439, in 
_process_added_router
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     
self._router_added(router['id'], router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 340, in 
_router_added
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     ri = 
self._create_router(router_id, router)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 337, in 
_create_router
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     return 
legacy_router.LegacyRouter(*args, **kwargs)
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent   File 
"/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 61, in 
__init__
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent     
DEFAULT_ADDRESS_SCOPE: ADDRESS_SCOPE_MARK_IDS.pop()}
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent KeyError: 'pop 
from an empty set'
  2016-04-05 09:30:09.033 24216 ERROR neutron.agent.l3.agent
  2016-04-05 09:30:09.034 24216 DEBUG neutron.agent.l3.agent [-] Starting 
router update for e341e0e2-5089-46e9-91f9-2099a156b27f, action None, priority 1 
_process_router_update 
/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py:463
  2016-04-05 09:30:09.035 24216 DEBUG oslo_messaging._drivers.amqpdriver [-] 
CALL msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 exchange 'neutron' topic 
'q-l3-plugin' _send 
/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:454
  2016-04-05 09:30:09.417 24216 DEBUG oslo_messaging._drivers.amqpdriver [-] 
received reply msg_id: 6295fbe9cf2040d79c68f5c5f8b1e963 __call__ 
/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:302
  2016-04-05 09:30:09.418 24216 ERROR neutron.agent.l3.agent [-] Failed to 
process compatible router 'e341e0e2-5089-46e9-91f9-2099a156b27f'

  So agent is constantly resyncing (causing load on neutron server) and
  unable to handle new routers.

  I believe that set "ADDRESS_SCOPE_MARK_IDS = set(range(1024, 2048))"
  from router_info.py should not be agent global but it should be
  ADDRESS_SCOPE_MARK_IDS  per router. Or at least need to return values
  back to the set when router is deleted.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1566291/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp

[Yahoo-eng-team] [Bug 1566291] [NEW] L3 agent: at some point an agent becomes unable to handle new routers

Reply via email to