Thanks Bence! Based on your findings, the OpenStack VMT will consider this a class C2 report (vulnerability in a dependency) per our taxonomy:
https://security.openstack.org/vmt-process.html#report-taxonomy If new information comes to light, please update this bug and we'll revisit the classification. ** Information type changed from Public Security to Public ** Changed in: ossa Status: Incomplete => Won't Fix ** Tags added: security -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1930414 Title: Traffic leaked from dhcp port before vlan tag is applied Status in neutron: New Status in OpenStack Security Advisory: Won't Fix Bug description: This is a bug with potential security implications. I don't see a clear way to exploit it at the moment, but to err on the safe side, I'm opening this as private to the security team. Short summary: Using openvswitch-agent, traffic sent on some (at least dhcp) ports before ovs-agent applies the port's vlan tag can be seen and intercepted on ports from other networks on the same integration bridge. We observed this bug: * using vlan and vxlan networks * using the noop and openvswitch firewall drivers * on openstack versions mitaka, pike and master (commit 5a6f61af4a) The time window between the port's creation and ovs-agent applying its vlan tag is usually very short. We observed this bug in the wild on a heavily loaded host. However to make the reproduction reliable on lightly loaded systems I inserted a sleep() into ovs-agent's source (just before the port's vlan tag is set): $ git --no-pager format-patch --stdout 5a6f61af4a From 8389b3e8e5c60c81ff2bb262e3ae2e8aab73d3f5 Mon Sep 17 00:00:00 2001 From: Bence Romsics <[email protected]> Date: Mon, 31 May 2021 13:12:34 +0200 Subject: [PATCH] WIP Change-Id: Ibef4278a2f6a85f52a8ffa43caef6de38cbb40cb --- .../plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py | 1 + 1 file changed, 1 insertion(+) diff --git a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py index 2c209bd387..355584b325 100644 --- a/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py +++ b/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py @@ -1190,6 +1190,7 @@ class OVSNeutronAgent(l2population_rpc.L2populationRpcCallBackTunnelMixin, self.setup_arp_spoofing_protection(self.int_br, port, port_detail) if cur_tag != lvm.vlan: + time.sleep(3000) self.int_br.set_db_attribute( "Port", port.port_name, "tag", lvm.vlan) · --· 2.31.0 We discovered the bug by the following procedure: * a test environment created multiple neutron nets in short succession with the exact same ipv6 subnet ranges * therefore neutron selected the exact same ipv6 address for the subnet's dhcp port * the host running the dhcp-agent and the ovs-agent was heavily loaded * we observed that many (when ovs-agent is made slow enough, then all but one) of these networks' services relying on the dhcp port's address were unavailable * because duplicate address detection (DAD) for the ipv6 dhcp port address failed * we believe DAD failed because we have some temporary crosstalk between the dhcp namespaces of different networks * we believe that this bug is not ipv6 specific, only the default DAD of ipv6 helped us discover it Exact reproduction steps: $ date --iso-8601=s 2021-05-31T13:10:14+00:00 # when the ovs-agent is slow enough, even 2 networks are sufficient $ for i in {1..5} do openstack network create xnet$i openstack subnet create xsubnet$i-v6 --ip-version 6 --network xnet$i --subnet-range 2001:db8::/32 done # for the record $ openstack subnet list -f value -c Name -c ID | egrep xsubnet 01d614da-820b-418d-8fa4-71952713f0ad xsubnet5-v6 72158e8e-5059-4abb-98a4-5adc9e4ef39c xsubnet2-v6 8f263143-a69b-4c42-b74c-6f30aca7b19d xsubnet4-v6 9ab4159e-12f8-44ed-8947-35a56b62eaf8 xsubnet1-v6 d4ed53e2-7b70-43d7-bd9f-d45f006a8179 xsubnet3-v6 # note that all dhcp ports got the same ip $ openstack port list --device-owner network:dhcp -f value -c id -c mac_address -c fixed_ips | egrep 2001:db8:: 130855be-ead1-40bb-9ca0-5336428aa74b fa:16:3e:24:76:41 [{'subnet_id': '01d614da-820b-418d-8fa4-71952713f0ad', 'ip_address': '2001:db8::1'}] 19fcabfd-f32a-40ea-b68e-ced41f394822 fa:16:3e:43:80:fe [{'subnet_id': '9ab4159e-12f8-44ed-8947-35a56b62eaf8', 'ip_address': '2001:db8::1'}] 46963dbd-c844-4986-a07f-fb78adbd95e9 fa:16:3e:4f:23:bf [{'subnet_id': '72158e8e-5059-4abb-98a4-5adc9e4ef39c', 'ip_address': '2001:db8::1'}] b8bf2fb1-d52a-41af-90bb-01aa23529015 fa:16:3e:90:40:8e [{'subnet_id': 'd4ed53e2-7b70-43d7-bd9f-d45f006a8179', 'ip_address': '2001:db8::1'}] ba67f2c0-c714-45fd-aec8-7233ba379dfa fa:16:3e:35:10:8d [{'subnet_id': '8f263143-a69b-4c42-b74c-6f30aca7b19d', 'ip_address': '2001:db8::1'}] # all but one dhcp port (and by the way metadata) addresses are in DAD failed mode $ for net in $( openstack network list -f value -c Name -c ID | awk '/ xnet/ { print $1 }' ) ; do sudo ip netns exec qdhcp-$net ip a ; done | egrep '(link/ether|inet6 (2001:db8::|fe80::a9fe:a9fe))' link/ether fa:16:3e:90:40:8e brd ff:ff:ff:ff:ff:ff inet6 2001:db8::1/32 scope global dadfailed tentative· inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative· link/ether fa:16:3e:4f:23:bf brd ff:ff:ff:ff:ff:ff inet6 2001:db8::1/32 scope global dadfailed tentative· inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative· link/ether fa:16:3e:24:76:41 brd ff:ff:ff:ff:ff:ff inet6 2001:db8::1/32 scope global dadfailed tentative· inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative· link/ether fa:16:3e:35:10:8d brd ff:ff:ff:ff:ff:ff inet6 2001:db8::1/32 scope global dadfailed tentative· inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative· link/ether fa:16:3e:43:80:fe brd ff:ff:ff:ff:ff:ff inet6 2001:db8::1/32 scope global· inet6 fe80::a9fe:a9fe/64 scope link· # dmesg also shows the DAD failures # man dmesg: Be aware that the timestamp could be inaccurate! $ LC_TIME=en_US sudo dmesg -T [snip] [Mon May 31 13:10:10 2021] device tap19fcabfd-f3 entered promiscuous mode [Mon May 31 13:10:15 2021] device tap46963dbd-c8 entered promiscuous mode [Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:15 2021] IPv6: tap46963dbd-c8: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:18 2021] device tapb8bf2fb1-d5 entered promiscuous mode [Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:20 2021] IPv6: tapb8bf2fb1-d5: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:22 2021] device tapba67f2c0-c7 entered promiscuous mode [Mon May 31 13:10:23 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:24 2021] IPv6: tapba67f2c0-c7: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:26 2021] device tap130855be-ea entered promiscuous mode [Mon May 31 13:10:27 2021] IPv6: tap130855be-ea: IPv6 duplicate address 2001:db8::1 used by fa:16:3e:43:80:fe detected! [Mon May 31 13:10:28 2021] IPv6: tap130855be-ea: IPv6 duplicate address fe80::a9fe:a9fe used by fa:16:3e:43:80:fe detected! # while there were no errors in ovs-agent $ sudo LC_TIME=en_US journalctl -u devstack@q-agt -S '2021-05-31 13:10:14' | egrep ERROR [empty] # clean up $ openstack network list | awk '/xnet/ { print $2 }' | xargs -r openstack network delete In a bit of further analysis I only created two networks and ran tcpdump on the first's dhcp port, while creating the second: $ openstack network create xnet0 $ openstack subnet create xsubnet0-v6 --ip-version 6 --network xnet0 --subnet-range 2001:db8::/32 # run tcpdump on the 1st net's dhcp port $ sudo ip netns exec qdhcp-$( openstack network show -f value -c id xnet0 ) tcpdump -n -vvv -i tapcaa92c34-53 # create the 2nd net while tcpdump is already running $ openstack network create xnet1 $ openstack subnet create xsubnet1-v6 --ip-version 6 --network xnet1 --subnet-range 2001:db8::/32 # the 2nd net's dhcp port's mac address $ openstack port list --device-owner network:dhcp --network xnet1 -f value -c mac_address fa:16:3e:0d:be:aa # tcpdump's capture contains packets with the 2nd net's dhcp port's mac address tcpdump: listening on tapcaa92c34-53, link-type EN10MB (Ethernet), capture size 262144 bytes ^C14:24:14.541893 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 1 group record(s) [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 14:24:14.929686 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 110: (hlim 1, next-header Options (0) payload length: 56) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 2 group record(s) [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 14:24:14.985673 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 14:24:14.993930 fa:16:3e:0d:be:aa > 33:33:ff:0d:be:aa, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff0d:beaa: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::f816:3eff:fe0d:beaa unknown option (14), length 8 (1):· 0x0000: bfdb db94 d695 14:24:15.209668 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 14:24:15.858999 fa:16:3e:0d:be:aa > 33:33:ff:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has 2001:db8::1 unknown option (14), length 8 (1):· 0x0000: 6e67 6a99 e72e 14:24:15.859815 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2001:db8::1 > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2001:db8::1, Flags [override] destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb 14:24:15.860188 fa:16:3e:0d:be:aa > 33:33:ff:fe:a9:fe, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:fffe:a9fe: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::a9fe:a9fe unknown option (14), length 8 (1):· 0x0000: 11e1 9aab 157a 14:24:15.860731 fa:16:3e:6a:ed:cb > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::a9fe:a9fe > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is fe80::a9fe:a9fe, Flags [override] destination link-address option (2), length 8 (1): fa:16:3e:6a:ed:cb 14:24:16.017855 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 14:24:16.137879 fa:16:3e:0d:be:aa > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 130: (hlim 1, next-header Options (0) payload length: 76) fe80::f816:3eff:fe0d:beaa > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener report v2, 3 group record(s) [gaddr ff02::1:ff00:1 to_ex, 0 source(s)] [gaddr ff02::1:fffe:a9fe to_ex, 0 source(s)] [gaddr ff02::1:ff0d:beaa to_ex, 0 source(s)] 11 packets captured 11 packets received by filter 0 packets dropped by kernel At the moment I did not test yet: * whether the leaked traffic can be intercepted on a vm port * whether a vm port can similarly leak traffic * whether ovs-agent can be attacked to intentionally slow it down To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1930414/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
