Public bug reported:
During the processing of a router by the L3 agent in
`agent_mode={dvr|dvr_snat}`, we create a floatingip_agent_gateway port.
The code in the L3-agent [0] and the neutron-server [1] assumes that there
should be only one such port per L3-agent per network. However, this is not the
case, and the parallel creation of two routers in the same external network can
result in the creation of two floatingip_agent_gateway ports attached to a
single L3-agent. The code [1] in the Neutron server is evidently susceptible to
a race condition.
- Firstly, only one port will actually be used, leading to the waste of an
additional IP address in the external network.
- Secondly, depending on the order in which the neutron-server returns these
ports, we may encounter an error in router processing by the L3-agent, with a
following retry attempt.
I deployed DevStack from the master branch on a <b>single<b> node and
reproduced the issue:
[root@devstack0 ~]# openstack router create r1 --external-gateway public &
openstack router create r2 --external-gateway public
[root@devstack0 ~]# openstack port list --network public --device-owner
network:floatingip_agent_gateway
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP
Addresses | Status |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| 309ed611-d4f1-4521-9b94-c202484cd1ac | | fa:16:3e:80:29:fe |
ip_address='10.20.30.41', subnet_id='dc701931-ad94-4558-b078-69288cbb2190' |
ACTIVE |
| ef195ebc-2f1a-4ede-8900-89bd8092062e | | fa:16:3e:f0:60:2c |
ip_address='10.20.30.58', subnet_id='dc701931-ad94-4558-b078-69288cbb2190' |
ACTIVE |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
[root@devstack0 ~]#
Relevant neutron-server logs:
Aug 21 05:07:55 devstack0 neutron-server[127666]: INFO neutron.db.l3_dvr_db
[None req-9b817aa9-748d-4666-8979-814c55737443 None None] Floating IP Agent
Gateway port for network dad747c6-c234-41e3-ae27-c9602b81fbd2 does not exist on
host devstack0. Creating one.
Aug 21 05:07:55 devstack0 neutron-server[127665]: INFO neutron.db.l3_dvr_db
[None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Agent
Gateway port for network dad747c6-c234-41e3-ae27-c9602b81fbd2 does not exist on
host devstack0. Creating one.
Aug 21 05:07:55 devstack0 neutron-server[127665]: DEBUG neutron.db.l3_dvr_db
[None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Gateway
port agent binding for network dad747c6-c234-41e3-ae27-c9602b81fbd2 already
exists on host devstack0. Probably it was just created by other worker.
{{(pid=127665) create_fip_agent_gw_port_if_not_exists
/opt/stack/neutron/neutron/db/l3_dvr_db.py:1201}}
...
Aug 21 05:07:55 devstack0 neutron-server[127665]: DEBUG neutron.db.l3_dvr_db
[None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Agent
Gateway port {'id': 'ef195ebc-2f1a-4ede-8900-89bd8092062e', 'name': '',
'network_id': 'dad747c6-c234-41e3-ae27-c9602b81fbd2', 'tenant_id': '',
'mac_address': 'fa:16:3e:f0:60:2c', 'admin_state_up': True, 'status': 'DOWN',
'device_id': 'c54e1e96-0e13-4ee1-94a9-3c355a377589', 'device_owner':
'network:floatingip_agent_gateway', 'standard_attr_id': 96, 'fixed_ips':
[{'subnet_id': 'dc701931-ad94-4558-b078-69288cbb2190', 'ip_address':
'10.20.30.58'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [],
'security_groups': [], 'description': '', 'binding:vnic_type': 'normal',
'binding:profile': {}, 'binding:host_id': 'devstack0', 'binding:vif_type':
'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True,
'ovs_hybrid_plug': False, 'datapath_type': 'system', 'bridge_name': 'br-int'},
'port_security_enabled': False, 'tags': [],
'created_at': '2024-08-21T09:07:55Z', 'updated_at': '2024-08-21T09:07:55Z',
'revision_number': 1, 'project_id': ''} created for the destination host:
devstack0 {{(pid=127665) create_fip_agent_gw_port_if_not_exists
/opt/stack/neutron/neutron/db/l3_dvr_db.py:1227}}
Aug 21 05:07:55 devstack0 neutron-server[127666]: DEBUG neutron.db.l3_dvr_db
[None req-9b817aa9-748d-4666-8979-814c55737443 None None] Floating IP Agent
Gateway port {'id': '309ed611-d4f1-4521-9b94-c202484cd1ac', 'name': '',
'network_id': 'dad747c6-c234-41e3-ae27-c9602b81fbd2', 'tenant_id': '',
'mac_address': 'fa:16:3e:80:29:fe', 'admin_state_up': True, 'status': 'DOWN',
'device_id': 'c54e1e96-0e13-4ee1-94a9-3c355a377589', 'device_owner':
'network:floatingip_agent_gateway', 'standard_attr_id': 95, 'fixed_ips':
[{'subnet_id': 'dc701931-ad94-4558-b078-69288cbb2190', 'ip_address':
'10.20.30.41'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [],
'security_groups': [], 'description': '', 'binding:vnic_type': 'normal',
'binding:profile': {}, 'binding:host_id': 'devstack0', 'binding:vif_type':
'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True,
'ovs_hybrid_plug': False, 'datapath_type': 'system', 'bridge_name': 'br-int'},
'port_security_enabled': False, 'tags': [],
'created_at': '2024-08-21T09:07:55Z', 'updated_at': '2024-08-21T09:07:55Z',
'revision_number': 1, 'project_id': ''} created for the destination host:
devstack0 {{(pid=127666) create_fip_agent_gw_port_if_not_exists
/opt/stack/neutron/neutron/db/l3_dvr_db.py:1227}}
[0]
https://opendev.org/openstack/neutron/src/commit/0807c94dc9843fff318c21d1f6f7b8838f948f5f/neutron/agent/l3/dvr_local_router.py#L835-L839
[1]
https://opendev.org/openstack/neutron/src/commit/0807c94dc9843fff318c21d1f6f7b8838f948f5f/neutron/db/l3_dvr_db.py#L1193-L1222
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2077532
Title:
Two floatingip_agent_gateways are created which are bound to the same
agent.
Status in neutron:
New
Bug description:
During the processing of a router by the L3 agent in
`agent_mode={dvr|dvr_snat}`, we create a floatingip_agent_gateway port.
The code in the L3-agent [0] and the neutron-server [1] assumes that there
should be only one such port per L3-agent per network. However, this is not the
case, and the parallel creation of two routers in the same external network can
result in the creation of two floatingip_agent_gateway ports attached to a
single L3-agent. The code [1] in the Neutron server is evidently susceptible to
a race condition.
- Firstly, only one port will actually be used, leading to the waste of an
additional IP address in the external network.
- Secondly, depending on the order in which the neutron-server returns these
ports, we may encounter an error in router processing by the L3-agent, with a
following retry attempt.
I deployed DevStack from the master branch on a <b>single<b> node and
reproduced the issue:
[root@devstack0 ~]# openstack router create r1 --external-gateway public &
openstack router create r2 --external-gateway public
[root@devstack0 ~]# openstack port list --network public --device-owner
network:floatingip_agent_gateway
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP
Addresses | Status |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| 309ed611-d4f1-4521-9b94-c202484cd1ac | | fa:16:3e:80:29:fe |
ip_address='10.20.30.41', subnet_id='dc701931-ad94-4558-b078-69288cbb2190' |
ACTIVE |
| ef195ebc-2f1a-4ede-8900-89bd8092062e | | fa:16:3e:f0:60:2c |
ip_address='10.20.30.58', subnet_id='dc701931-ad94-4558-b078-69288cbb2190' |
ACTIVE |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
[root@devstack0 ~]#
Relevant neutron-server logs:
Aug 21 05:07:55 devstack0 neutron-server[127666]: INFO neutron.db.l3_dvr_db
[None req-9b817aa9-748d-4666-8979-814c55737443 None None] Floating IP Agent
Gateway port for network dad747c6-c234-41e3-ae27-c9602b81fbd2 does not exist on
host devstack0. Creating one.
Aug 21 05:07:55 devstack0 neutron-server[127665]: INFO neutron.db.l3_dvr_db
[None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Agent
Gateway port for network dad747c6-c234-41e3-ae27-c9602b81fbd2 does not exist on
host devstack0. Creating one.
Aug 21 05:07:55 devstack0 neutron-server[127665]: DEBUG neutron.db.l3_dvr_db
[None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Gateway
port agent binding for network dad747c6-c234-41e3-ae27-c9602b81fbd2 already
exists on host devstack0. Probably it was just created by other worker.
{{(pid=127665) create_fip_agent_gw_port_if_not_exists
/opt/stack/neutron/neutron/db/l3_dvr_db.py:1201}}
...
Aug 21 05:07:55 devstack0 neutron-server[127665]: DEBUG neutron.db.l3_dvr_db
[None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Agent
Gateway port {'id': 'ef195ebc-2f1a-4ede-8900-89bd8092062e', 'name': '',
'network_id': 'dad747c6-c234-41e3-ae27-c9602b81fbd2', 'tenant_id': '',
'mac_address': 'fa:16:3e:f0:60:2c', 'admin_state_up': True, 'status': 'DOWN',
'device_id': 'c54e1e96-0e13-4ee1-94a9-3c355a377589', 'device_owner':
'network:floatingip_agent_gateway', 'standard_attr_id': 96, 'fixed_ips':
[{'subnet_id': 'dc701931-ad94-4558-b078-69288cbb2190', 'ip_address':
'10.20.30.58'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [],
'security_groups': [], 'description': '', 'binding:vnic_type': 'normal',
'binding:profile': {}, 'binding:host_id': 'devstack0', 'binding:vif_type':
'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True,
'ovs_hybrid_plug': False, 'datapath_type': 'system', 'bridge_name': 'br-int'},
'port_security_enabled': False, 'tags': [
], 'created_at': '2024-08-21T09:07:55Z', 'updated_at': '2024-08-21T09:07:55Z',
'revision_number': 1, 'project_id': ''} created for the destination host:
devstack0 {{(pid=127665) create_fip_agent_gw_port_if_not_exists
/opt/stack/neutron/neutron/db/l3_dvr_db.py:1227}}
Aug 21 05:07:55 devstack0 neutron-server[127666]: DEBUG neutron.db.l3_dvr_db
[None req-9b817aa9-748d-4666-8979-814c55737443 None None] Floating IP Agent
Gateway port {'id': '309ed611-d4f1-4521-9b94-c202484cd1ac', 'name': '',
'network_id': 'dad747c6-c234-41e3-ae27-c9602b81fbd2', 'tenant_id': '',
'mac_address': 'fa:16:3e:80:29:fe', 'admin_state_up': True, 'status': 'DOWN',
'device_id': 'c54e1e96-0e13-4ee1-94a9-3c355a377589', 'device_owner':
'network:floatingip_agent_gateway', 'standard_attr_id': 95, 'fixed_ips':
[{'subnet_id': 'dc701931-ad94-4558-b078-69288cbb2190', 'ip_address':
'10.20.30.41'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [],
'security_groups': [], 'description': '', 'binding:vnic_type': 'normal',
'binding:profile': {}, 'binding:host_id': 'devstack0', 'binding:vif_type':
'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True,
'ovs_hybrid_plug': False, 'datapath_type': 'system', 'bridge_name': 'br-int'},
'port_security_enabled': False, 'tags': [
], 'created_at': '2024-08-21T09:07:55Z', 'updated_at': '2024-08-21T09:07:55Z',
'revision_number': 1, 'project_id': ''} created for the destination host:
devstack0 {{(pid=127666) create_fip_agent_gw_port_if_not_exists
/opt/stack/neutron/neutron/db/l3_dvr_db.py:1227}}
[0]
https://opendev.org/openstack/neutron/src/commit/0807c94dc9843fff318c21d1f6f7b8838f948f5f/neutron/agent/l3/dvr_local_router.py#L835-L839
[1]
https://opendev.org/openstack/neutron/src/commit/0807c94dc9843fff318c21d1f6f7b8838f948f5f/neutron/db/l3_dvr_db.py#L1193-L1222
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2077532/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
