Findings: with an ASG of 20 nodes, one of them failed to run the user-data when running cloud-init v18.3-3
Entire user-data script for reference: #!/bin/bash echo "I was here" | tee -a /tmp/uds-logging I was also able to run `cloud-init analyze show` on the failing (and succeeding) nodes. The failing nodes seem to skip entirely the "modules-final" stage Successful run: ----------------------------------------------------------------------------------------------------------------------------------- $ sudo cloud-init analyze show -- Boot Record 01 -- The total time elapsed since completing an event is printed after the "@" character. The time the event takes is printed after the "+" character. Starting stage: single Starting stage: init-local |`->cache invalid in datasource: DataSourceEc2 @18647.94500s +05.06200s |`->no local data found from DataSourceEc2Local @18653.00900s +00.03200s Finished stage: (init-local) 05.14900 seconds Starting stage: init-network |`->no cache found @18653.56600s +00.00000s |`->found network data from DataSourceEc2 @18653.57000s +00.12600s |`->setting up datasource @18653.73800s +00.00000s |`->reading and applying user-data @18653.74700s +00.00300s |`->reading and applying vendor-data @18653.75000s +00.00000s |`->activating datasource @18653.76600s +00.00100s |`->config-migrator ran successfully @18653.82700s +00.00100s |`->config-seed_random ran successfully @18653.82800s +00.00100s |`->config-growpart ran successfully @18653.82900s +00.04800s |`->config-bootcmd ran successfully @18653.87800s +00.00000s |`->config-write-files ran successfully @18653.87800s +00.00100s |`->config-growpart ran successfully @18653.87900s +00.02400s |`->config-resizefs ran successfully @18653.90300s +00.02000s |`->config-disk_setup ran successfully @18653.92400s +00.00100s |`->config-mounts ran successfully @18653.92500s +00.00500s |`->config-set_hostname ran successfully @18653.93000s +00.00100s |`->config-update_hostname ran successfully @18653.93100s +00.00100s |`->config-update_etc_hosts ran successfully @18653.93200s +00.00400s |`->config-ca-certs ran successfully @18653.93600s +00.00100s |`->config-rsyslog ran successfully @18653.93700s +00.00100s |`->config-users-groups ran successfully @18653.93800s +00.00100s |`->config-ssh ran successfully @18653.94300s +00.22000s Finished stage: (init-network) 00.60700 seconds Starting stage: modules-config |`->config-emit_upstart ran successfully @18655.03900s +00.00100s |`->config-ssh-import-id ran successfully @18655.04100s +00.00100s |`->config-locale ran successfully @18655.04200s +00.00100s |`->config-set-passwords ran successfully @18655.04400s +00.00100s |`->config-grub-dpkg ran successfully @18655.04500s +00.23700s |`->config-apt-pipelining ran successfully @18655.28300s +00.00300s |`->config-apt-configure ran successfully @18655.28600s +00.09200s |`->config-ntp ran successfully @18655.37900s +00.00200s |`->config-timezone ran successfully @18655.38100s +00.00200s |`->config-disable-ec2-metadata ran successfully @18655.38300s +00.00100s |`->config-runcmd ran successfully @18655.38400s +00.00100s |`->config-byobu ran successfully @18655.38600s +00.00100s Finished stage: (modules-config) 00.39000 seconds Starting stage: modules-final |`->config-package-update-upgrade-install ran successfully @18658.53600s +00.00200s |`->config-fan ran successfully @18658.53800s +00.00100s |`->config-puppet ran successfully @18658.54000s +00.00100s |`->config-chef ran successfully @18658.54100s +00.00100s |`->config-salt-minion ran successfully @18658.54200s +00.00100s |`->config-mcollective ran successfully @18658.54300s +00.00100s |`->config-rightscale_userdata ran successfully @18658.54500s +00.00400s |`->config-scripts-vendor ran successfully @18658.55000s +00.00200s |`->config-scripts-per-once previously ran @18658.55300s +00.00000s |`->config-scripts-per-boot ran successfully @18658.55300s +00.00100s |`->config-scripts-per-instance ran successfully @18658.55400s +00.00100s |`->config-scripts-user ran successfully @18658.55600s +00.00800s |`->config-ssh-authkey-fingerprints ran successfully @18658.56500s +00.00900s |`->config-keys-to-console ran successfully @18658.57500s +00.08700s |`->config-phone-home ran successfully @18658.66200s +00.00200s |`->config-final-message ran successfully @18658.66500s +00.00700s |`->config-power-state-change ran successfully @18658.67300s +00.00100s Finished stage: (modules-final) 00.24300 seconds Total Time: 6.38900 seconds 1 boot records analyzed ----------------------------------------------------------------------------------------------------------------------------------- Failed run: ----------------------------------------------------------------------------------------------------------------------------------- $ sudo cloud-init analyze show -- Boot Record 01 -- The total time elapsed since completing an event is printed after the "@" character. The time the event takes is printed after the "+" character. Starting stage: single Starting stage: init-local |`->cache invalid in datasource: DataSourceEc2 @18619.79200s +05.04800s |`->no local data found from DataSourceEc2Local @18624.84200s +00.01800s Finished stage: (init-local) 05.09500 seconds Starting stage: init-network |`->no cache found @18625.41900s +00.00000s |`->found network data from DataSourceEc2 @18625.42300s +00.11500s |`->setting up datasource @18625.58000s +00.00000s |`->reading and applying user-data @18625.59000s +00.00300s |`->reading and applying vendor-data @18625.59300s +00.00000s |`->activating datasource @18625.60900s +00.00100s |`->config-migrator ran successfully @18625.65900s +00.00100s |`->config-seed_random ran successfully @18625.66000s +00.00100s |`->config-growpart ran successfully @18625.66100s +00.03700s |`->config-bootcmd ran successfully @18625.69800s +00.00000s |`->config-write-files ran successfully @18625.69900s +00.00000s |`->config-growpart ran successfully @18625.70000s +00.02400s |`->config-resizefs ran successfully @18625.72400s +00.02000s |`->config-disk_setup ran successfully @18625.74500s +00.00100s |`->config-mounts ran successfully @18625.74700s +00.00200s |`->config-set_hostname ran successfully @18625.75000s +00.00100s |`->config-update_hostname ran successfully @18625.75100s +00.00100s |`->config-update_etc_hosts ran successfully @18625.75300s +00.00400s |`->config-ca-certs ran successfully @18625.75700s +00.00100s |`->config-rsyslog ran successfully @18625.75800s +00.00000s |`->config-users-groups ran successfully @18625.75900s +00.00100s |`->config-ssh ran successfully @18625.76000s +00.20700s Finished stage: (init-network) 00.55900 seconds Starting stage: modules-config |`->config-emit_upstart ran successfully @18626.96000s +00.00000s |`->config-ssh-import-id ran successfully @18626.96100s +00.00200s |`->config-locale ran successfully @18626.96300s +00.00200s |`->config-set-passwords ran successfully @18626.96500s +00.00100s |`->config-grub-dpkg ran successfully @18626.96600s +00.24200s |`->config-apt-pipelining ran successfully @18627.20800s +00.00200s |`->config-apt-configure ran successfully @18627.21200s +00.08000s |`->config-ntp ran successfully @18627.29200s +00.00100s |`->config-timezone ran successfully @18627.29400s +00.00100s |`->config-disable-ec2-metadata ran successfully @18627.29500s +00.00100s |`->config-runcmd ran successfully @18627.29600s +00.00100s |`->config-byobu ran successfully @18627.29700s +00.00100s Finished stage: (modules-config) 00.43600 seconds Total Time: 6.09000 seconds 1 boot records analyzed ----------------------------------------------------------------------------------------------------------------------------------- How can I triage this further? Thank you for your time and assistance; Jason On Tue, Dec 4, 2018 at 11:12 AM Jason Price <[email protected]> wrote: > > On Tue, Dec 4, 2018 at 3:06 AM Thomas Goirand <[email protected]> wrote: > > > > On 12/4/18 4:00 AM, Jason Price wrote: > > > Thank you for any assistance, and please let me know how I can help. > > Could you try with the latest cloud-init from Sid? > > Alright, I've begun testing with the new build. I'll respond back if > that helps, but it'll take some time to gain the confidence
