Public bug reported: This might be a duplicate of: https://bugs.launchpad.net/neutron/+bug/1798577
however, I see the same issue with floating IPs and DVR, not just SNAT. This breaks the FWaaSv2 service, resulting in no filtering. Deployment uses Kolla Ansible 8.0.1 (Stein) on CentOS, which installs neutron-fwaas from here: http://tarballs.openstack.org/neutron-fwaas/neutron-fwaas-15.0.0.tar.gz DVR is being used, but the issue appears to exist on both distributed routers and centralized routers on the network nodes (both qrouter and snat namespaces). I am attaching a small bash script that creates a firewall rule, policy, and group to block TCP port 25 outbound. When this didn't work (outbound SMTP traffic was not blocked), I looked at the iptables rules on the network node and compute nodes. These are the interfaces in the qrouter namespace of a test router that has the issue with fwaasv2: 2: rfp-3f6273be-2@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether be:8b:2c:40:dc:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 169.254.125.2/31 scope global rfp-3f6273be-2 valid_lft forever preferred_lft forever inet6 fe80::bc8b:2cff:fe40:dce5/64 scope link valid_lft forever preferred_lft forever 1118: qr-a418f15b-fa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:6f:fe:2c brd ff:ff:ff:ff:ff:ff inet 192.168.99.254/24 brd 192.168.99.255 scope global qr-a418f15b-fa valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe6f:fe2c/64 scope link valid_lft forever preferred_lft forever The is the iptables config (iptables -S), with some notes injected: -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -N neutron-filter-top -N neutron-l3-agent-FORWARD -N neutron-l3-agent-INPUT -N neutron-l3-agent-OUTPUT -N neutron-l3-agent-accepted -N neutron-l3-agent-dropped -N neutron-l3-agent-fwaas-defau -N neutron-l3-agent-iv4ff1b0489 -N neutron-l3-agent-local -N neutron-l3-agent-ov4ff1b0489 -N neutron-l3-agent-rejected -N neutron-l3-agent-scope -A INPUT -j neutron-l3-agent-INPUT -A FORWARD -j neutron-filter-top -A FORWARD -j neutron-l3-agent-FORWARD -A OUTPUT -j neutron-filter-top -A OUTPUT -j neutron-l3-agent-OUTPUT -A neutron-filter-top -j neutron-l3-agent-local -A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope # NOTE: These 4 rules have the wrong interface - note that "a418f15b-f" is # the name of the qr interface, NOT the rfp interface (see above interface list): -A neutron-l3-agent-FORWARD -o rfp-a418f15b-f -j neutron-l3-agent-iv4ff1b0489 -A neutron-l3-agent-FORWARD -i rfp-a418f15b-f -j neutron-l3-agent-ov4ff1b0489 -A neutron-l3-agent-FORWARD -o rfp-a418f15b-f -j neutron-l3-agent-fwaas-defau -A neutron-l3-agent-FORWARD -i rfp-a418f15b-f -j neutron-l3-agent-fwaas-defau -A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT -A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP -A neutron-l3-agent-accepted -j ACCEPT -A neutron-l3-agent-dropped -j DROP -A neutron-l3-agent-fwaas-defau -j neutron-l3-agent-dropped -A neutron-l3-agent-iv4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped -A neutron-l3-agent-iv4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT -A neutron-l3-agent-iv4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped -A neutron-l3-agent-ov4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped -A neutron-l3-agent-ov4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT -A neutron-l3-agent-ov4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped -A neutron-l3-agent-rejected -j REJECT --reject-with icmp-port-unreachable # NOTE: These two rules use the correct interface names: -A neutron-l3-agent-scope -o rfp-3f6273be-2 -m mark ! --mark 0x4000000/0xffff0000 -j DROP -A neutron-l3-agent-scope -o qr-a418f15b-fa -m mark ! --mark 0x4000000/0xffff0000 -j DROP This obviously breaks the FWaaSv2 pretty severely (iptables simply ignores the rules with the incorrect interface names). The same issue occurs in the SNAT namespace... The SNAT namespace interfaces: 1115: ha-8632c48f-29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9150 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:ee:dd:91 brd ff:ff:ff:ff:ff:ff inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-8632c48f-29 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feee:dd91/64 scope link valid_lft forever preferred_lft forever 1143: sg-8f9c085a-be: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:3a:31:f8 brd ff:ff:ff:ff:ff:ff 1155: qg-8e38e9f5-28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:f9:3a:d3 brd ff:ff:ff:ff:ff:ff The iptables (iptables -S) output - with notes injected: -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -N neutron-filter-top -N neutron-l3-agent-FORWARD -N neutron-l3-agent-INPUT -N neutron-l3-agent-OUTPUT -N neutron-l3-agent-accepted -N neutron-l3-agent-dropped -N neutron-l3-agent-fwaas-defau -N neutron-l3-agent-iv4ff1b0489 -N neutron-l3-agent-local -N neutron-l3-agent-ov4ff1b0489 -N neutron-l3-agent-rejected -N neutron-l3-agent-scope -A INPUT -j neutron-l3-agent-INPUT -A FORWARD -j neutron-filter-top -A FORWARD -j neutron-l3-agent-FORWARD -A OUTPUT -j neutron-filter-top -A OUTPUT -j neutron-l3-agent-OUTPUT -A neutron-filter-top -j neutron-l3-agent-local -A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope # NOTE: These 4 rules have the wrong interface - note that "a418f15b-fa" is # the name of the qr interface, NOT the sg interface. It is actually the "rfp" # interface from the qrouter namespace! -A neutron-l3-agent-FORWARD -o sg-a418f15b-fa -j neutron-l3-agent-iv4ff1b0489 -A neutron-l3-agent-FORWARD -i sg-a418f15b-fa -j neutron-l3-agent-ov4ff1b0489 -A neutron-l3-agent-FORWARD -o sg-a418f15b-fa -j neutron-l3-agent-fwaas-defau -A neutron-l3-agent-FORWARD -i sg-a418f15b-fa -j neutron-l3-agent-fwaas-defau -A neutron-l3-agent-accepted -j ACCEPT -A neutron-l3-agent-dropped -j DROP -A neutron-l3-agent-fwaas-defau -j neutron-l3-agent-dropped -A neutron-l3-agent-iv4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped -A neutron-l3-agent-iv4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT -A neutron-l3-agent-iv4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped -A neutron-l3-agent-ov4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped -A neutron-l3-agent-ov4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT -A neutron-l3-agent-ov4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped -A neutron-l3-agent-rejected -j REJECT --reject-with icmp-port-unreachable # NOTE: This rule has the correct interface name: -A neutron-l3-agent-scope -o sg-8f9c085a-be -m mark ! --mark 0x4000000/0xffff0000 -j DROP FWaaSv2 is a pretty critical component, and I haven't found a patch that has been made after Stein (maybe I just didn't find it though). Eric ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1851194 Title: FWaaSv2 configures iptables with invalid port name Status in neutron: New Bug description: This might be a duplicate of: https://bugs.launchpad.net/neutron/+bug/1798577 however, I see the same issue with floating IPs and DVR, not just SNAT. This breaks the FWaaSv2 service, resulting in no filtering. Deployment uses Kolla Ansible 8.0.1 (Stein) on CentOS, which installs neutron-fwaas from here: http://tarballs.openstack.org/neutron-fwaas/neutron-fwaas-15.0.0.tar.gz DVR is being used, but the issue appears to exist on both distributed routers and centralized routers on the network nodes (both qrouter and snat namespaces). I am attaching a small bash script that creates a firewall rule, policy, and group to block TCP port 25 outbound. When this didn't work (outbound SMTP traffic was not blocked), I looked at the iptables rules on the network node and compute nodes. These are the interfaces in the qrouter namespace of a test router that has the issue with fwaasv2: 2: rfp-3f6273be-2@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether be:8b:2c:40:dc:e5 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 169.254.125.2/31 scope global rfp-3f6273be-2 valid_lft forever preferred_lft forever inet6 fe80::bc8b:2cff:fe40:dce5/64 scope link valid_lft forever preferred_lft forever 1118: qr-a418f15b-fa: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:6f:fe:2c brd ff:ff:ff:ff:ff:ff inet 192.168.99.254/24 brd 192.168.99.255 scope global qr-a418f15b-fa valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:fe6f:fe2c/64 scope link valid_lft forever preferred_lft forever The is the iptables config (iptables -S), with some notes injected: -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -N neutron-filter-top -N neutron-l3-agent-FORWARD -N neutron-l3-agent-INPUT -N neutron-l3-agent-OUTPUT -N neutron-l3-agent-accepted -N neutron-l3-agent-dropped -N neutron-l3-agent-fwaas-defau -N neutron-l3-agent-iv4ff1b0489 -N neutron-l3-agent-local -N neutron-l3-agent-ov4ff1b0489 -N neutron-l3-agent-rejected -N neutron-l3-agent-scope -A INPUT -j neutron-l3-agent-INPUT -A FORWARD -j neutron-filter-top -A FORWARD -j neutron-l3-agent-FORWARD -A OUTPUT -j neutron-filter-top -A OUTPUT -j neutron-l3-agent-OUTPUT -A neutron-filter-top -j neutron-l3-agent-local -A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope # NOTE: These 4 rules have the wrong interface - note that "a418f15b-f" is # the name of the qr interface, NOT the rfp interface (see above interface list): -A neutron-l3-agent-FORWARD -o rfp-a418f15b-f -j neutron-l3-agent-iv4ff1b0489 -A neutron-l3-agent-FORWARD -i rfp-a418f15b-f -j neutron-l3-agent-ov4ff1b0489 -A neutron-l3-agent-FORWARD -o rfp-a418f15b-f -j neutron-l3-agent-fwaas-defau -A neutron-l3-agent-FORWARD -i rfp-a418f15b-f -j neutron-l3-agent-fwaas-defau -A neutron-l3-agent-INPUT -m mark --mark 0x1/0xffff -j ACCEPT -A neutron-l3-agent-INPUT -p tcp -m tcp --dport 9697 -j DROP -A neutron-l3-agent-accepted -j ACCEPT -A neutron-l3-agent-dropped -j DROP -A neutron-l3-agent-fwaas-defau -j neutron-l3-agent-dropped -A neutron-l3-agent-iv4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped -A neutron-l3-agent-iv4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT -A neutron-l3-agent-iv4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped -A neutron-l3-agent-ov4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped -A neutron-l3-agent-ov4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT -A neutron-l3-agent-ov4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped -A neutron-l3-agent-rejected -j REJECT --reject-with icmp-port-unreachable # NOTE: These two rules use the correct interface names: -A neutron-l3-agent-scope -o rfp-3f6273be-2 -m mark ! --mark 0x4000000/0xffff0000 -j DROP -A neutron-l3-agent-scope -o qr-a418f15b-fa -m mark ! --mark 0x4000000/0xffff0000 -j DROP This obviously breaks the FWaaSv2 pretty severely (iptables simply ignores the rules with the incorrect interface names). The same issue occurs in the SNAT namespace... The SNAT namespace interfaces: 1115: ha-8632c48f-29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9150 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:ee:dd:91 brd ff:ff:ff:ff:ff:ff inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-8632c48f-29 valid_lft forever preferred_lft forever inet6 fe80::f816:3eff:feee:dd91/64 scope link valid_lft forever preferred_lft forever 1143: sg-8f9c085a-be: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:3a:31:f8 brd ff:ff:ff:ff:ff:ff 1155: qg-8e38e9f5-28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether fa:16:3e:f9:3a:d3 brd ff:ff:ff:ff:ff:ff The iptables (iptables -S) output - with notes injected: -P INPUT ACCEPT -P FORWARD ACCEPT -P OUTPUT ACCEPT -N neutron-filter-top -N neutron-l3-agent-FORWARD -N neutron-l3-agent-INPUT -N neutron-l3-agent-OUTPUT -N neutron-l3-agent-accepted -N neutron-l3-agent-dropped -N neutron-l3-agent-fwaas-defau -N neutron-l3-agent-iv4ff1b0489 -N neutron-l3-agent-local -N neutron-l3-agent-ov4ff1b0489 -N neutron-l3-agent-rejected -N neutron-l3-agent-scope -A INPUT -j neutron-l3-agent-INPUT -A FORWARD -j neutron-filter-top -A FORWARD -j neutron-l3-agent-FORWARD -A OUTPUT -j neutron-filter-top -A OUTPUT -j neutron-l3-agent-OUTPUT -A neutron-filter-top -j neutron-l3-agent-local -A neutron-l3-agent-FORWARD -j neutron-l3-agent-scope # NOTE: These 4 rules have the wrong interface - note that "a418f15b-fa" is # the name of the qr interface, NOT the sg interface. It is actually the "rfp" # interface from the qrouter namespace! -A neutron-l3-agent-FORWARD -o sg-a418f15b-fa -j neutron-l3-agent-iv4ff1b0489 -A neutron-l3-agent-FORWARD -i sg-a418f15b-fa -j neutron-l3-agent-ov4ff1b0489 -A neutron-l3-agent-FORWARD -o sg-a418f15b-fa -j neutron-l3-agent-fwaas-defau -A neutron-l3-agent-FORWARD -i sg-a418f15b-fa -j neutron-l3-agent-fwaas-defau -A neutron-l3-agent-accepted -j ACCEPT -A neutron-l3-agent-dropped -j DROP -A neutron-l3-agent-fwaas-defau -j neutron-l3-agent-dropped -A neutron-l3-agent-iv4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped -A neutron-l3-agent-iv4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT -A neutron-l3-agent-iv4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped -A neutron-l3-agent-ov4ff1b0489 -m state --state INVALID -j neutron-l3-agent-dropped -A neutron-l3-agent-ov4ff1b0489 -m state --state RELATED,ESTABLISHED -j ACCEPT -A neutron-l3-agent-ov4ff1b0489 -p tcp -m tcp --dport 25 -j neutron-l3-agent-dropped -A neutron-l3-agent-rejected -j REJECT --reject-with icmp-port-unreachable # NOTE: This rule has the correct interface name: -A neutron-l3-agent-scope -o sg-8f9c085a-be -m mark ! --mark 0x4000000/0xffff0000 -j DROP FWaaSv2 is a pretty critical component, and I haven't found a patch that has been made after Stein (maybe I just didn't find it though). Eric To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1851194/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
