** Changed in: neutron Status: New => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1790706
Title: Additional metadata service endpoints on OpenStack accessible Status in neutron: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: Note: I'm reporting this on behalf of our partner SAP. While the bug is about Newton, one of our neutron developers believes this may still be valid for newer versions: "The bug might be still valid in upstream, since there are no specific case where they are filtering based on the IP 169.254.169.254, since they are passing the same port as such." # Setup: OpenStack Newton with `force_metadata = true` on all network nodes Kubernetes Gardener setup (seed+shoot) on OpenStack # Detail description from the hacker simulation: By running a nmap -sn … scan (ping scan) we discovered several endpoints in the shoot network (apart from the nodes that can be seen from `kubectl --kubeconfig myshoot.kubeconfig get nodes`). We noticed that some of the endpoints also serve meta and user data on port 80, i.e. the metadata service is not only available from the well-known metadata service IP (http://169.254.169.254/…, https://docs.openstack.org/nova/latest/user/metadata-service.html) but also from those other addresses. In our test the endpoints were 10.250.0.2-7. We learned that the endpoints probably are the OpenStack DHCP nodes, i.e. every OpenStack DHCP endpoint appears to also serve the metadata. While the accessibility of the metadata service is a known problem, this situation is “worse” (compared to e.g. Gardener seed and shoot clusters on AWS) for the following reasons: 1. If a network policy is applied to block access from cluster payloads to the metadata service, it’s not enough to block well-known `169.254.169.254` but it must also block all access to all other existing endpoints. How can the definite set of endpoints be determined? Are they guaranteed to not change during the lifetime of a cluster? 2. If the metadata service is only accessible via 169.254.169.254, the known `kubectl proxy` issue (details can be shared if needed) cannot be used to get access to the metadata service, as the link-local 169.254.0.0/16 address range is not allowed by the Kubernetes API server as an endpoint address. But for example 10.250… is allowed, i.e. a shoot user on OpenStack can use the attack to get access to the metadata service in the seed network. The fact that no fix is in sight for the `kubectl proxy` issue and it might not be patchable poses an additional risk regarding 2. We will try to follow up on that with the Kubernetes security team once again. # Detail information: Due to the “force_metadata” setting the DHCP namespaces are exposing the metadata service: # ip netns exec qdhcp-54ad9fe0-2ce5-4083-a32b-ca744e806d1f netstat -tulpen Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 0 1934832519 54198/python tcp 0 0 10.222.0.3:53 0.0.0.0:* LISTEN 0 1934920972 54135/dnsmasq tcp 0 0 169.254.169.254:53 0.0.0.0:* LISTEN 0 1934920970 54135/dnsmasq tcp 0 0 fe80::f816:3eff:fe01:53 :::* LISTEN 198 1934909191 54135/dnsmasq udp 0 0 10.222.0.3:53 0.0.0.0:* 0 1934920971 54135/dnsmasq udp 0 0 169.254.169.254:53 0.0.0.0:* 0 1934920969 54135/dnsmasq udp 0 0 0.0.0.0:67 0.0.0.0:* 0 1934920966 54135/dnsmasq udp 0 0 fe80::f816:3eff:fe01:53 :::* 198 1934909190 54135/dnsmasq The problem is that the metadata proxy is listening to 0.0.0.0:80 instead of 169.254.169.254:80. This let the metadata service respond also to DHCP ip addresses which cannot be blocked easily. This fix mitigated the problem: --- neutron.org/agent/metadata/namespace_proxy.py 2018-08-31 12:42:25.901681939 +0000 +++ neutron/agent/metadata/namespace_proxy.py 2018-08-31 12:43:17.541826180 +0000 @@ -130,7 +130,7 @@ self.router_id) proxy = wsgi.Server('neutron-network-metadata-proxy', num_threads=self.proxy_threads) - proxy.start(handler, self.port) + proxy.start(handler, self.port, '169.254.169.254') # Drop privileges after port bind super(ProxyDaemon, self).run() To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1790706/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp