Public bug reported: Context ======= OpenStack Stein (but master seems affected by this as well). OVS based deployment. L3 routers in DVR and HA mode. One big public "external/public" network (with subnets like /21 or /22) used by instances and router external gateways.
Problem description =================== When adding a gateway on a router in HA+DVR, neutron api may send a lot of RPC messages toward L3 agents, depending on the size of the subnet used for the gateway. How to reproduce ================ Add a gateway on a router: $ openstack router set --external-gateway Ext-Net router-arnaud On neutron server, in logs (in DEBUG): Notify agent at l3_agent.hostxyz We see this line for all l3 agents having a port in Ext-Net subnet (which can be huge, like 1k). Then, all agents are doing another RPC call (sync_routers) which is ending on neutron-rpc with this log line: Sync routers for ids [abc] Behing the Sync router, a big SQL request is done [1] When 1k requests like this are done, on each router update, the database is killed by too much SQL requests to do. [1] https://github.com/openstack/neutron/blob/stable/stein/neutron/db/l3_dvrscheduler_db.py#L363 ** Affects: neutron Importance: Undecided Assignee: Arnaud Morin (arnaud-morin) Status: In Progress ** Changed in: neutron Assignee: (unassigned) => Arnaud Morin (arnaud-morin) ** Summary changed: - [scale] Router update is killing database + [scale] Setting a gateway on router is killing database -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1992950 Title: [scale] Setting a gateway on router is killing database Status in neutron: In Progress Bug description: Context ======= OpenStack Stein (but master seems affected by this as well). OVS based deployment. L3 routers in DVR and HA mode. One big public "external/public" network (with subnets like /21 or /22) used by instances and router external gateways. Problem description =================== When adding a gateway on a router in HA+DVR, neutron api may send a lot of RPC messages toward L3 agents, depending on the size of the subnet used for the gateway. How to reproduce ================ Add a gateway on a router: $ openstack router set --external-gateway Ext-Net router-arnaud On neutron server, in logs (in DEBUG): Notify agent at l3_agent.hostxyz We see this line for all l3 agents having a port in Ext-Net subnet (which can be huge, like 1k). Then, all agents are doing another RPC call (sync_routers) which is ending on neutron-rpc with this log line: Sync routers for ids [abc] Behing the Sync router, a big SQL request is done [1] When 1k requests like this are done, on each router update, the database is killed by too much SQL requests to do. [1] https://github.com/openstack/neutron/blob/stable/stein/neutron/db/l3_dvrscheduler_db.py#L363 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1992950/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
