Public bug reported: [Impact]
- It is not possible for multiple ovn-backed neutron ports with port security enabled to share a VIP among themselves - The proposed design is to have two firewalls with port security enabled (each in their own respective availability zone so that the overall architecture is highly available) share the subnet's gateway IP by assigning it to each port's allowed address pair list so that it can float between them in an ACTIVE_STANDBY setup. Below is the high level design +-----------------------------------+-----------------------------------+ | AZ1 | AZ2 | +-----------------------------------+-----------------------------------+ | Server: az1-fw | Server: az2-fw | | Port: az1-fw-port | Port: az2-fw-port | | IP: 172.16.11.254 | IP: 172.16.11.251 | | MAC: fa:16:3e:1f:bd:4d | MAC: fa:16:3e:4b:b4:39 | | Allowed Address Pair | Allowed Address Pair | | IP: 172.16.11.1 | IP: 172.16.11.1 | | MAC: 00:00:5e:00:01:01 | MAC: 00:00:5e:00:01:01 | +-----------------------------------+-----------------------------------+ - The expected behaviour is that when one appliance fails, OVN's switching would generate new flows to route to the new machine that's garping the VIP (since both machines are allowed to claim the VIP), but this does not happen - When port security is disabled the new flows are learned and the failover happens successfully - In this case there is no keepalived / VRRP implemented. Both machines statically own the VIP and only one machine garps at a time. The intended detection and port forwarding is expected to behave purely based on switching and MAC learning. However, VRRP workfloads are also likely impacted (potentially [1] is such a case) - There are several reasons that the expected behaviour doesn't work based on bot OVN the arguments neutron uses to invoke OVN. Firstly, the presence of port security on a logical switch port historically disabled FDB learning to preserve an authoritative mapping. However, this was recently fixed in OVN at [2] since port security filtering is applied before FDB learning and so there isn't a risk of learning a flow that would otherwise be disallowed. A secondary issue is that OpenFlow does not support identical flows that only differ based on an output port. Essentially, the two LSPs will race to create the first flow, and the second will never be able eligible for learning. There is another prerequisite on OVN LSPs in order to make the port eligible for FDB learning and that is the presence of the unknown address token, which allows for OVN to deliver packets that are destined for MACs that aren't explicitly listed in the addresses field. This leads into the problems on the neutron side. Firstly, when port security is enabled neutron specifically creates the LSP without the unknown address token (making it ineligible for learning). Secondly, when an allowed address pair is present, neutron statically programs the allowed address' MAC into the LSP's addresses field. This is a static declaration of ownership and so when the same AAP is present on multiple ports multiple conflicting OpenFlows are created and only one of them can be actuated. [Test Plan] Strictly speaking, the AZs aren't necessary to manifest the issue so that will be omitted from the test plan to make it simpler (but it was mentioned to highlight the practical use case) 1. Deploy openstack (eg. devstack or charmed openstack) 2. Create a network and subnet openstack network create case-repro-l2-nw openstack subnet create case-repro-l2-subnet \ --network case-repro-l2-nw \ --subnet-range 172.16.11.0/24 \ --no-dhcp 3. Create ports on that subnet and assign AAP AZ1 FW ========= openstack port create az1-fw-port \ --network case-repro-l2-nw \ --fixed-ip subnet=case-repro-l2-subnet,ip-address=172.16.11.254 \ --mac-address fa:16:3e:1f:bd:4d openstack port set az1-fw-port \ --allowed-address ip-address=172.16.11.1,mac-address=00:00:5e:00:01:01 AZ2 FW ========= openstack port create az2-fw-port \ --network case-repro-l2-nw \ --fixed-ip subnet=case-repro-l2-subnet,ip-address=172.16.11.251 \ --mac-address fa:16:3e:4b:b4:39 openstack port set az2-fw-port \ --allowed-address ip-address=172.16.11.1,mac-address=00:00:5e:00:01:01 Neutral client (a machine that doesn't have the VIP on any interface) ============== openstack port create client1-port \ --network case-repro-l2-nw \ --fixed-ip subnet=case-repro-l2-subnet,ip-address=172.16.11.250 4. Create servers (the interfaces on the private network are useful so we can configure and change the other interfaces without connectivity loss) # These commands assume the existence of resolute (Ubuntu 26.04) and cirros images in glance, the presence of an arbitrary network called private, and a keypair called mykey AZ1 FW ========= openstack server create az1-fw \ --image resolute \ --flavor m1.medium \ --network private \ --port az1-fw-port \ --key-name mykey AZ2 FW ========= openstack server create az2-fw \ --image resolute \ --flavor m1.medium \ --network private \ --port az2-fw-port \ --key-name mykey Client ========= openstack server create client \ --image cirros-0.6.3-x86_64-disk \ --flavor m1.small \ --network private \ --port client1-port \ --key-name mykey 5. Ssh into the machines and configure their interfaces sudo ip netns exec <private-nw-namespace-uuid> ssh ~/.ssh/id_ed25519 ubuntu@<azX-fw-private-ip> eg. in my case the private network has the subnet 10.0.0.0/24 sudo ip netns exec ovnmeta-99b5c2a7-eff9-407a-ac9b-7c0c8c61e31f ssh -i ~/.ssh/id_ed25519 [email protected] # AZ1 sudo ip netns exec ovnmeta-99b5c2a7-eff9-407a-ac9b-7c0c8c61e31f ssh -i ~/.ssh/id_ed25519 [email protected] # AZ2 sudo ip netns exec ovnmeta-99b5c2a7-eff9-407a-ac9b-7c0c8c61e31f ssh -i ~/.ssh/id_ed25519 [email protected] # client AZ1 FW ========= sudo ip addr add 172.16.11.254/24 dev ens4 sudo ip link set ens4 up sudo ip link add vip0 link ens4 type macvlan mode bridge sudo ip link set vip0 address 00:00:5e:00:01:01 # fa:16:3e:aa:bb:cc # 00:00:5e:00:01:01 sudo ip addr add 172.16.11.1/32 dev vip0 sudo ip link set vip0 up # Make sure packets with src .1 go out vip0. In particular this ensures that inbound packets destined for .1 are routed back out through vip0 and pick up the VIP MAC sudo ip rule add from 172.16.11.1/32 table 100 sudo ip route add 172.16.11.0/24 dev vip0 src 172.16.11.1 table 100 The interfaces should look like this ip a 3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc pfifo_fast state UP group default qlen 1000 link/ether fa:16:3e:1f:bd:4d brd ff:ff:ff:ff:ff:ff altname enp0s4 altname enxfa163e1fbd4d inet 172.16.11.254/24 brd 172.16.11.255 scope global ens4 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe1f:bd4d/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 4: vip0@ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1442 qdisc noqueue state UP group default qlen 1000 link/ether 00:00:5e:00:01:01 brd ff:ff:ff:ff:ff:ff inet 172.16.11.1/32 scope global vip0 valid_lft forever preferred_lft forever inet6 fe80::200:5eff:fe00:101/64 scope link proto kernel_ll valid_lft forever preferred_lft forever AZ2 FW ========= sudo ip addr add 172.16.11.254/24 dev ens4 sudo ip link set ens4 up sudo ip link add vip0 link ens4 type macvlan mode bridge sudo ip link set vip0 address 00:00:5e:00:01:01 # fa:16:3e:aa:bb:cc # 00:00:5e:00:01:01 sudo ip addr add 172.16.11.1/32 dev vip0 sudo ip link set vip0 up # Make sure packets with src .1 go out vip0. In particular this ensures that inbound packets destined for .1 are routed back out through vip0 and pick up the VIP MAC sudo ip rule add from 172.16.11.1/32 table 100 sudo ip route add 172.16.11.0/24 dev vip0 src 172.16.11.1 table 100 Client ========= sudo ip addr add 172.16.11.250/24 dev eth1 sudo ip link set eth1 up 6. Check generated flows Observe that there are two identical flows that only differ by outport (and chassis) sudo ovn-sbctl lflow-list | grep -E \ '172\.16\.11\.1|00:00:5e:00:01:01|172\.16\.11\.250' table=0 (ls_in_check_port_sec), priority=110 , match=(((ip4 && icmp4.type == 3 && icmp4.code == 4) || (ip6 && icmp6.type == 2 && icmp6.code == 0)) && eth.src == 00:00:5e:00:01:01 && outport == "1f9370cb-ffb6-464e-8c32-c1b0a8046c01" && !is_chassis_resident("1f9370cb-ffb6-464e-8c32-c1b0a8046c01") && flags.tunnel_rx == 1), action=(outport <-> inport; next;) table=0 (ls_in_check_port_sec), priority=110 , match=(((ip4 && icmp4.type == 3 && icmp4.code == 4) || (ip6 && icmp6.type == 2 && icmp6.code == 0)) && eth.src == 00:00:5e:00:01:01 && outport == "499b4447-4549-43ca-bad5-6d04ee8c633a" && !is_chassis_resident("499b4447-4549-43ca-bad5-6d04ee8c633a") && flags.tunnel_rx == 1), action=(outport <-> inport; next;) openstack port list +--------------------------------------+---------------+-------------------+------------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+---------------+-------------------+------------------------------------------------------------------------------+--------+ | 1f9370cb-ffb6-464e-8c32-c1b0a8046c01 | az1-fw-port1 | fa:16:3e:1f:bd:4d | ip_address='172.16.11.254', subnet_id='c6ff137d-b3ff-45ec-8791-7b9bee741e9f' | ACTIVE | | 499b4447-4549-43ca-bad5-6d04ee8c633a | az2-fw-port1 | fa:16:3e:4b:b4:39 | ip_address='172.16.11.251', subnet_id='c6ff137d-b3ff-45ec-8791-7b9bee741e9f' | ACTIVE | +--------------------------------------+---------------+-------------------+------------------------------------------------------------------------------+--------+ The first matching flow is destined for AZ1 7. Check connectivity On AZ1 (.254) ping AZ2 (.251) - This works On AZ1 (.254) ping client (.250) - This works On AZ2 (.251) ping AZ1 (.254) - This works On AZ2 (.251) ping client (.250) - This works On client (.250) ping AZ1 (.254) - This works On client (.250) ping AZ2 (.251) - This works On client (.250) ping VIP (.1) - This works Running tcpdump on AZ1 shows traffic 8. Test failover In this case on AZ1 (since it actively owns the VIP) sudo ip link set down vip0 ping .1 from the client and observe that return traffic stops. Wait a few minutes and packet loss continues. 9. Inspect the OVN LSPs eg. az1-fw-port sudo ovn-nbctl list Logical_Switch_Port 1f9370cb-ffb6-464e-8c32-c1b0a8046c01 _uuid : 8176b021-07d4-4776-b06d-d73969810547 addresses : ["00:00:5e:00:01:01", "fa:16:3e:1f:bd:4d 172.16.11.254"] dhcpv4_options : [] dhcpv6_options : [] dynamic_addresses : [] enabled : true external_ids : {"neutron:cidrs"="172.16.11.254/24", "neutron:device_id"="4fce1268-4163-4272-bd5d-eff7be322b01", "neutron:device_owner"="compute:nova", "neutron:host_id"=devstack-aap-ovn, "neutron:mtu"="", "neutron:network_name"=neutron-d5272c4f-2c9c-4009-a610-625601d07622, "neutron:port_capabilities"="", "neutron:port_name"=az1-fw-port1, "neutron:project_id"="8e3602474a474b889fad27bde017b7d3", "neutron:revision_number"="14", "neutron:security_group_ids"="bcec972f-a0f0-48a3-8f14-d18c636e5d29", "neutron:subnet_pool_addr_scope4"="", "neutron:subnet_pool_addr_scope6"="", "neutron:vnic_type"=normal} ha_chassis_group : [] health_checks : [] mirror_rules : [] name : "1f9370cb-ffb6-464e-8c32-c1b0a8046c01" options : {requested-chassis=devstack-aap-ovn} parent_name : [] peer : [] port_security : ["00:00:5e:00:01:01 172.16.11.1", "VRRPv3 fa:16:3e:1f:bd:4d 00:00:5e:00:01:01 172.16.11.1", "fa:16:3e:1f:bd:4d 172.16.11.254"] tag : [] tag_request : [] type : "" up : true The key detail is that address has the VIP MAC statically programmed and no unknown address token 10. Modify the OVN LSPs' addresses to remove the static MAC and add the unknown address token AZ1 FW ========= sudo ovn-nbctl lsp-set-addresses 1f9370cb-ffb6-464e-8c32-c1b0a8046c01 "fa:16:3e:1f:bd:4d 172.16.11.254" unknown AZ2 FW ========= sudo ovn-nbctl lsp-set-addresses 499b4447-4549-43ca-bad5-6d04ee8c633a "fa:16:3e:4b:b4:39 172.16.11.251" unknown Traffic should start to flow 11. Reconfigure vip0 and bring it up on az1-fw as in step 5 This doesn't change anything yet, packets still arrive on az2-fw (since it actively owns the port) 12. Bring down vip0 on az2-fw Momentarily some packets will be dropped due to the arp cache timeout (anecdotally this seemed to be 15s but was not strictly timed). Then packets start arriving on the other VM [1] Potentially similar bug with VRRP managing VIP ownership https://bugs.launchpad.net/ubuntu/+source/ovn/+bug/2166984 [2] https://github.com/ovn-org/ovn/commit/873ac74eb7553737f57f8dc4f79fb6408524296b#diff-7ee66c4f1536ac84dc5bbff1b8312e2eef24b974b3e48a5c5c2bcfdf2eb8f3ce ** Affects: neutron Importance: Undecided Status: New ** Affects: neutron (Ubuntu) Importance: Undecided Status: New ** Affects: ovn (Ubuntu) Importance: Undecided Status: New ** Affects: neutron (Ubuntu Jammy) Importance: Undecided Status: New ** Affects: ovn (Ubuntu Jammy) Importance: Undecided Status: New ** Affects: neutron (Ubuntu Noble) Importance: Undecided Status: New ** Affects: ovn (Ubuntu Noble) Importance: Undecided Status: New ** Affects: neutron (Ubuntu Resolute) Importance: Undecided Status: New ** Affects: ovn (Ubuntu Resolute) Importance: Undecided Status: New ** Affects: neutron (Ubuntu Stonking) Importance: Undecided Status: New ** Affects: ovn (Ubuntu Stonking) Importance: Undecided Status: New ** Affects: neutron (Ubuntu Tenacious) Importance: Undecided Status: New ** Affects: ovn (Ubuntu Tenacious) Importance: Undecided Status: New ** Also affects: ovn (Ubuntu) Importance: Undecided Status: New ** Also affects: ovn (Ubuntu Jammy) Importance: Undecided Status: New ** Also affects: ovn (Ubuntu Resolute) Importance: Undecided Status: New ** Also affects: ovn (Ubuntu Stonking) Importance: Undecided Status: New ** Also affects: ovn (Ubuntu Noble) Importance: Undecided Status: New ** Also affects: ovn (Ubuntu Tenacious) Importance: Undecided Status: New ** Also affects: neutron (Ubuntu) Importance: Undecided Status: New ** Summary changed: - FDB Learning of AAP MACs does not work on OVN-backed neutron ports with port security enabled + FDB Learning of AAP MACs does not work on OVN-backed Neutron Ports when Port Security is Enabled -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2170987 Title: FDB Learning of AAP MACs does not work on OVN-backed Neutron Ports when Port Security is Enabled To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2170987/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
