Reviewed: https://review.opendev.org/c/openstack/neutron/+/900647 Committed: https://opendev.org/openstack/neutron/commit/f2a3020cf0a46dbd896c5f7b4b4f6643d32a6b4a Submitter: "Zuul (22348)" Branch: master
commit f2a3020cf0a46dbd896c5f7b4b4f6643d32a6b4a Author: Luis Tomas Bolivar <[email protected]> Date: Mon Nov 13 16:42:51 2023 +0100 Ensure ovn loadbalancer FIPs are centralized upon neutron restarts When neutron server restarts the mac address for NAT entries related to ovn-lb FIPs gets re-added, distributing the traffic that should be centralized and therefore breaking the connectivity. This happens due to the port being down. This patch is ensuring the MAC entry is only being readded in case the port is UP Closes-Bug: #2042938 Change-Id: I6203009750a4e589eeb808f842cb522d61476179 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2042938 Title: [OVN] Neutron Service Restart Disrupts Octavia OVN Load Balancer Floating IP Status in neutron: Fix Released Bug description: Description: Following the Neutron service restart, the Octavia OVN load balancer equipped with floating IPs stops processing traffic (DVR FIP is enabled) Affected Versions: - Neutron: Stable/Yoga (commit 03ed5578f7) - Octavia: Stable/Yoga (commit 83955125) - OVN: Version 22.12 (package ovn22.12-22.12.0-34.el9s.x86_64) Details: It looks like the 'external_mac' attribute for the Load Balancer's floating IP is changed when neutron is restarted. It start working again when one does detach/attach FIP to LB VIP. here is a before neutron server restart: ovn-nbctl find NAT type=dnat_and_snat external_ip=128.130.194.85 _uuid : 438dff1d-3245-491c-a534-0a33e395ab2c allowed_ext_ips : [] exempted_ext_ips : [] external_ids : {"neutron:fip_external_mac"="fa:16:3e:cb:96:e7", "neutron:fip_id"="36fb8531-3a33-453e-be46-64599976aa0e", "neutron:fip_network_id"="1f115e41-cc7f-4bb5-8493-995d035277dc", "neutron:fip_port_id"="9ef678e5-add1-47f5-ac7c-a23f4ee753bb", "neutron:revision_number"="50", "neutron:router_name"=neutron-96bd0fd6-bc77-4b17-a5b4-e79c20b448ce} external_ip : "128.130.194.85" external_mac : [] external_port_range : "" gateway_port : [] logical_ip : "192.168.100.124" logical_port : "9ef678e5-add1-47f5-ac7c-a23f4ee753bb" options : {} type : dnat_and_snat after neutron server restart: ovn-nbctl find NAT type=dnat_and_snat external_ip=128.130.194.8 _uuid : 028a9017-544b-49e6-82cb-86cb1bdfab54 allowed_ext_ips : [] exempted_ext_ips : [] external_ids : {"neutron:fip_external_mac"="fa:16:3e:f1:08:95", "neutron:fip_id"="3bbd5870-54ba-4941-a85a-29691d8b7aae", "neutron:fip_network_id"="1f115e41-cc7f-4bb5-8493-995d035277dc", "neutron:fip_port_id"="f2791464-91b6-4ace-8255-5e4aaf9785e0", "neutron:revision_number"="30", "neutron:router_name"=neutron-6d32d3de-38cc-4175-9764-5bdd1efbc20e} external_ip : "128.130.194.8" external_mac : "fa:16:3e:f1:08:95" external_port_range : "" gateway_port : [] logical_ip : "192.168.100.213" logical_port : "f2791464-91b6-4ace-8255-5e4aaf9785e0" options : {} type : dnat_and_snat here are the debug logs entries for this FIP when neutron restarts: 2023-11-07 12:46:26.165 28 DEBUG ovsdbapp.backend.ovs_idl.event [req-4ac48a1f-12b8-4d33-a05a-5ca856fe1895 - - - - -] Matched CREATE: LogicalSwitchPortCreateDownEvent(events=('create',), table='Logical_Switch_Port', conditions=(('up', '=', False),), old_conditions=None), priority=20 to row=Logical_Switch_Port(port_security=['fa:16:3e:54:5e:8c 192.168.3.236'], addresses=[], type=, dhcpv4_options=[<ovs.db.idl.Row object at 0x7f1a36a172b0>], name=a803471b-0254-4f66-911e-03a143c874b8, up=[False], options={'requested-chassis': ''}, ha_chassis_group=[], external_ids={'neutron:cidrs': '192.168.3.236/24', 'neutron:device_id': '', 'neutron:device_owner': '', 'neutron:network_name': 'neutron- eb3ed2db-3c79-4e78-92ac-5653f9abb094', 'neutron:port_fip': '128.130.194.113', 'neutron:port_name': 'ovn-lb- vip-c60d029a-7625-4a96-837d-46ba95f56507', 'neutron:project_id': 'ea03412a87bc426eae8b7badd15aa7a8', 'neutron:revision_number': '1', 'neutron:security_group_ids': 'f558ec6a-fcb7-4297-8737-11a1507099b0', 'neutron:subnet_pool_addr_scope4': '', 'neutron:subnet_pool_addr_scope6': ''}, dynamic_addresses=[], tag=[], parent_name=[], mirror_rules=[], tag_request=[], enabled=[True], dhcpv6_options=[]) old= matches /var/lib/kolla/venv/lib/python3.9/site- packages/ovsdbapp/backend/ovs_idl/event.py:43 2023-11-07 12:46:26.209 28 DEBUG ovsdbapp.backend.ovs_idl.event [req-4ac48a1f-12b8-4d33-a05a-5ca856fe1895 - - - - -] Matched CREATE: FIPAddDeleteEvent(events=('create', 'delete'), table='NAT', conditions=(('type', '=', 'dnat_and_snat'),), old_conditions=None), priority=20 to row=NAT(external_ids={'neutron:fip_external_mac': 'fa:16:3e:17:d1:2c', 'neutron:fip_id': '840d24da-368d-4851-9b19-131c8f882f81', 'neutron:fip_network_id': '1f115e41-cc7f-4bb5-8493-995d035277dc', 'neutron:fip_port_id': 'a803471b-0254-4f66-911e-03a143c874b8', 'neutron:revision_number': '8', 'neutron:router_name': 'neutron-83f83d78-b4a5-4fe0-8e32-352f6d941583'}, external_ip=128.130.194.113, allowed_ext_ips=[], external_port_range=, exempted_ext_ips=[], logical_ip=192.168.3.236, type=dnat_and_snat, external_mac=['fa:16:3e:17:d1:2c'], options={}, logical_port=['a803471b-0254-4f66-911e-03a143c874b8'], gateway_port=[]) old= matches /var/lib/kolla/venv/lib/python3.9/site- packages/ovsdbapp/backend/ovs_idl/event.py:43 2023-11-07 12:46:26.225 31 DEBUG ovsdbapp.backend.ovs_idl.event [req-985a5a2e-6838-45c1-9879-aa8bc0732d00 - - - - -] Matched CREATE: FIPAddDeleteEvent(events=('create', 'delete'), table='NAT', conditions=(('type', '=', 'dnat_and_snat'),), old_conditions=None), priority=20 to row=NAT(external_ids={'neutron:fip_external_mac': 'fa:16:3e:17:d1:2c', 'neutron:fip_id': '840d24da-368d-4851-9b19-131c8f882f81', 'neutron:fip_network_id': '1f115e41-cc7f-4bb5-8493-995d035277dc', 'neutron:fip_port_id': 'a803471b-0254-4f66-911e-03a143c874b8', 'neutron:revision_number': '8', 'neutron:router_name': 'neutron-83f83d78-b4a5-4fe0-8e32-352f6d941583'}, external_ip=128.130.194.113, allowed_ext_ips=[], external_port_range=, exempted_ext_ips=[], logical_ip=192.168.3.236, type=dnat_and_snat, external_mac=['fa:16:3e:17:d1:2c'], options={}, logical_port=['a803471b-0254-4f66-911e-03a143c874b8'], gateway_port=[]) old= matches /var/lib/kolla/venv/lib/python3.9/site- packages/ovsdbapp/backend/ovs_idl/event.py:43 2023-11-07 12:46:42.564 31 DEBUG oslo_concurrency.processutils [req-e1654d04-1e37-44e2-a10b-16fc937ba532 - - - - -] Running cmd (subprocess): ovsdb-client transact tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642 --timeout 180 ["OVN_Southbound", {"op": "delete", "table": "MAC_Binding", "where": [["ip", "==", "128.130.194.113"]]}] execute /var/lib/kolla/venv/lib/python3.9/site- packages/oslo_concurrency/processutils.py:384 2023-11-07 12:46:42.575 31 DEBUG oslo_concurrency.processutils [req-e1654d04-1e37-44e2-a10b-16fc937ba532 - - - - -] CMD "ovsdb-client transact tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642 --timeout 180 ["OVN_Southbound", {"op": "delete", "table": "MAC_Binding", "where": [["ip", "==", "128.130.194.113"]]}]" returned: 0 in 0.011s execute /var/lib/kolla/venv/lib/python3.9/site- packages/oslo_concurrency/processutils.py:422 2023-11-07 12:46:49.623 28 DEBUG oslo_concurrency.processutils [req-c8da7baa-f23d-4a4d-aa5f-307ac4957ee6 - - - - -] Running cmd (subprocess): ovsdb-client transact tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642 --timeout 180 ["OVN_Southbound", {"op": "delete", "table": "MAC_Binding", "where": [["ip", "==", "128.130.194.113"]]}] execute /var/lib/kolla/venv/lib/python3.9/site- packages/oslo_concurrency/processutils.py:384 2023-11-07 12:46:49.633 28 DEBUG oslo_concurrency.processutils [req-c8da7baa-f23d-4a4d-aa5f-307ac4957ee6 - - - - -] CMD "ovsdb-client transact tcp:10.20.1.11:6642,tcp:10.20.1.12:6642,tcp:10.20.1.13:6642 --timeout 180 ["OVN_Southbound", {"op": "delete", "table": "MAC_Binding", "where": [["ip", "==", "128.130.194.113"]]}]" returned: 0 in 0.010s execute /var/lib/kolla/venv/lib/python3.9/site- packages/oslo_concurrency/processutils.py:422 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2042938/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
