Public bug reported: cloud-init 18.4-7-g4652b196-0ubuntu1 on Azure does not retry IMDS if the service is not yet available.
Per configuration in the azure datasource, it should retry 4 times before failing, but logs only list one attempt at http://169.254.169.254/metadata/instance?api-version=2017-12-01 We can see this in the following excerpt of /var/log/cloud-init.log 2018-10-26 17:26:51,162 - url_helper.py[DEBUG]: [0/4] open 'http://169.254.169.254/metadata/instance?api-version=2017-12-01' with {'url': 'http://169.254.169.254/metadata/instance?api-version=2017-12-01', 'allow_redirects': True, 'method': 'GET', 'timeout': 1.0, 'headers': {'User-Agent': 'Cloud-Init/18.4-7-g4652b196-0ubuntu1', 'Metadata': 'true'}} configuration 2018-10-26 17:26:52,168 - DataSourceAzure.py[DEBUG]: Ignoring IMDS instance metadata: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /metadata/instance?api-version=2017-12-01 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f392a7c5668>, 'Connection to 169.254.169.254 timed out. (connect timeout=1.0)')) 2018-10-26 17:26:52,168 - util.py[DEBUG]: Crawl of Azure Instance Metadata Service (IMDS) took 1.007 seconds Only 1 attempt (out of 4) was attempted and then Azure datasource gave up after receiving a ConnectTimeoutError. At a later time on the instance, I was able to see that the IMDS service was available using the following python script: $ python3 -c 'from cloudinit.sources.DataSourceAzure import get_metadata_from_imds; print(get_metadata_from_imds("eth0", retries=2))'; {'compute': {'location': 'eastus2', 'name': 'my-c1', 'offer': 'UbuntuServer', 'osType': 'Linux', 'placementGroupId': '', 'platformFaultDomain': '0', 'platformUpdateDomain': '0', 'publisher': 'Canonical', 'resourceGroupName': 'srugroup1', 'sku': '18.10-DAILY', 'subscriptionId': '12aad61c-6de4-4e53-a6c6-5aff52a83777', 'tags': '', 'version': '18.10.201810230', 'vmId': 'ec22ac8c-f728-425f-852f- 5b86d0c9f2f7', 'vmScaleSetName': '', 'vmSize': 'Standard_DS1_v2', 'zone': ''}, 'network': {'interface': [{'ipv4': {'ipAddress': [{'privateIpAddress': '10.0.0.4', 'publicIpAddress': '40.70.24.192'}], 'subnet': [{'address': '10.0.0.0', 'prefix': '24'}]}, 'ipv6': {'ipAddress': []}, 'macAddress': '000D3A03582B'}]}} ** Affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1800223 Title: Azure: IMDS service not retried on timeout Status in cloud-init: New Bug description: cloud-init 18.4-7-g4652b196-0ubuntu1 on Azure does not retry IMDS if the service is not yet available. Per configuration in the azure datasource, it should retry 4 times before failing, but logs only list one attempt at http://169.254.169.254/metadata/instance?api-version=2017-12-01 We can see this in the following excerpt of /var/log/cloud-init.log 2018-10-26 17:26:51,162 - url_helper.py[DEBUG]: [0/4] open 'http://169.254.169.254/metadata/instance?api-version=2017-12-01' with {'url': 'http://169.254.169.254/metadata/instance?api-version=2017-12-01', 'allow_redirects': True, 'method': 'GET', 'timeout': 1.0, 'headers': {'User-Agent': 'Cloud-Init/18.4-7-g4652b196-0ubuntu1', 'Metadata': 'true'}} configuration 2018-10-26 17:26:52,168 - DataSourceAzure.py[DEBUG]: Ignoring IMDS instance metadata: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /metadata/instance?api-version=2017-12-01 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f392a7c5668>, 'Connection to 169.254.169.254 timed out. (connect timeout=1.0)')) 2018-10-26 17:26:52,168 - util.py[DEBUG]: Crawl of Azure Instance Metadata Service (IMDS) took 1.007 seconds Only 1 attempt (out of 4) was attempted and then Azure datasource gave up after receiving a ConnectTimeoutError. At a later time on the instance, I was able to see that the IMDS service was available using the following python script: $ python3 -c 'from cloudinit.sources.DataSourceAzure import get_metadata_from_imds; print(get_metadata_from_imds("eth0", retries=2))'; {'compute': {'location': 'eastus2', 'name': 'my-c1', 'offer': 'UbuntuServer', 'osType': 'Linux', 'placementGroupId': '', 'platformFaultDomain': '0', 'platformUpdateDomain': '0', 'publisher': 'Canonical', 'resourceGroupName': 'srugroup1', 'sku': '18.10-DAILY', 'subscriptionId': '12aad61c-6de4-4e53-a6c6-5aff52a83777', 'tags': '', 'version': '18.10.201810230', 'vmId': 'ec22ac8c-f728-425f-852f- 5b86d0c9f2f7', 'vmScaleSetName': '', 'vmSize': 'Standard_DS1_v2', 'zone': ''}, 'network': {'interface': [{'ipv4': {'ipAddress': [{'privateIpAddress': '10.0.0.4', 'publicIpAddress': '40.70.24.192'}], 'subnet': [{'address': '10.0.0.0', 'prefix': '24'}]}, 'ipv6': {'ipAddress': []}, 'macAddress': '000D3A03582B'}]}} To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1800223/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp