Public bug reported: I came across an issue today for a user that was experiencing issues connecting to metadata at 169.254.169.254. For a long time, cloud-init has had a fallback mechanism to that allowed it to contact the metadata service at http://<dhcp server ip>/latest/meta-data if http://169.254.169.254/latest/meta-data were unavailable, like so:
[ 157.574921] cloud-init[1313]: 2020-03-31 09:53:24,158 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5890>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))] [ 208.629083] cloud-init[1313]: 2020-03-31 09:54:15,214 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5350>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))] [ 226.639267] cloud-init[1313]: 2020-03-31 09:54:33,224 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5250>, 'Connection to 169.254.169.254 timed out. (connect timeout=17.0)'))] [ 227.640812] cloud-init[1313]: 2020-03-31 09:54:34,225 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds [ 227.651134] cloud-init[1313]: 2020-03-31 09:54:34,236 - url_helper.py[WARNING]: Calling 'http://10.19.48.2/latest/meta-data/instance-id' failed [0/120s]: request error [('Connection aborted.', error(111, 'Connection refused'))] [ 228.655226] cloud-init[1313]: 2020-03-31 09:54:35,240 - url_helper.py[WARNING]: Calling 'http://10.19.48.2/latest/meta-data/instance-id' failed [1/120s]: request error [('Connection aborted.', error(111, 'Connection refused'))] In this Stein environment, isolated metadata is enabled, and the qdhcp namespace has a listener at 169.254.169.254:80. Previous versions of Neutron had the listener on 0.0.0.0:80, which helped facilitate the fallback mechanism described above. The bug/patch where this was changed is here: [1] https://bugs.launchpad.net/neutron/+bug/1745618 Having this functionality back would be nice. Thoughts? ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1870228 Title: cloud-init metadata fallback broken Status in neutron: New Bug description: I came across an issue today for a user that was experiencing issues connecting to metadata at 169.254.169.254. For a long time, cloud-init has had a fallback mechanism to that allowed it to contact the metadata service at http://<dhcp server ip>/latest/meta-data if http://169.254.169.254/latest/meta-data were unavailable, like so: [ 157.574921] cloud-init[1313]: 2020-03-31 09:53:24,158 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [50/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5890>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))] [ 208.629083] cloud-init[1313]: 2020-03-31 09:54:15,214 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [101/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5350>, 'Connection to 169.254.169.254 timed out. (connect timeout=50.0)'))] [ 226.639267] cloud-init[1313]: 2020-03-31 09:54:33,224 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [119/120s]: request error [HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /2009-04-04/meta-data/instance-id (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.HTTPConnection object at 0x7fa9e07c5250>, 'Connection to 169.254.169.254 timed out. (connect timeout=17.0)'))] [ 227.640812] cloud-init[1313]: 2020-03-31 09:54:34,225 - DataSourceEc2.py[CRITICAL]: Giving up on md from ['http://169.254.169.254/2009-04-04/meta-data/instance-id'] after 120 seconds [ 227.651134] cloud-init[1313]: 2020-03-31 09:54:34,236 - url_helper.py[WARNING]: Calling 'http://10.19.48.2/latest/meta-data/instance-id' failed [0/120s]: request error [('Connection aborted.', error(111, 'Connection refused'))] [ 228.655226] cloud-init[1313]: 2020-03-31 09:54:35,240 - url_helper.py[WARNING]: Calling 'http://10.19.48.2/latest/meta-data/instance-id' failed [1/120s]: request error [('Connection aborted.', error(111, 'Connection refused'))] In this Stein environment, isolated metadata is enabled, and the qdhcp namespace has a listener at 169.254.169.254:80. Previous versions of Neutron had the listener on 0.0.0.0:80, which helped facilitate the fallback mechanism described above. The bug/patch where this was changed is here: [1] https://bugs.launchpad.net/neutron/+bug/1745618 Having this functionality back would be nice. Thoughts? To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1870228/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
