I'm removing the cloud-init tasks as pitti agrees the diagnosis seems sane.
@Steve, feel free to open a bug against cloud-init to use the builtin support, and we can do that independently of this. ** No longer affects: cloud-init (Ubuntu) -- You received this bug notification because you are a member of नेपाली भाषा समायोजकहरुको समूह, which is subscribed to Xenial. Matching subscriptions: Ubuntu 16.04 Bugs https://bugs.launchpad.net/bugs/1623570 Title: Azure: cannot start walinux agent (Transaction order is cyclic.) Status in walinuxagent package in Ubuntu: Fix Released Status in cloud-init source package in Xenial: Confirmed Status in walinuxagent source package in Xenial: Confirmed Bug description: http://pad.lv/1623570 ==== Begin SRU Template ==== [Impact] On Azure (and only on Azure), cloud-init utilizes the walinuxagent to do some of the instance initialization. A change to cloud-init to support installation of packages on a systemd system (bug 1576692), exposed an issue in the walinuxagent.service file. The end result is freshly booted instances on azure with the cloud-init fix and the old version of walinuxagent would fail to import ssh keys that were provided to the instance. [Test Case] * Start an instance on Azure * upgrade to cloud-init at 0.7.7-31-g65ace7b or better this is in both xenial-proposed and yakkety. * clean up, so it looks like a first boot. sudo sh -c 'rm -Rf /var/lib/cloud /var/log/cloud-init* /var/lib/waagent/*.crt' * At this point a reboot would cause failure, and you would see WARN messages in the logs. * upgrade walinuxagent with a fix (2.1.5-0ubuntu2 in yakkety) * reboot * ssh back in, * There should be no WARN in /var/log/cloud-init.log grep WARN /var/log/cloud-init.log * There should exist a file /var/lib/waagent/*.crt * walinuxagent.service should be running systemctl status walinuxagent.service Note, that in event of failure you can still get into the system by providing user-data that populates your ssh keys: #!/bin/sh ssh-import-id smoser [Regression Potential] Changing order of things in boot is scary. This is no different. This would in theory allow walinux-agent to start earlier in the boot. In practice, however, it is started by cloud-init.service so it was already running that early in the boot. ==== End SRU Template ==== When bringing up the Azure datasource in cloud-init.service, cloud-init tries 'service start walinuxagent'. That previously worked fine, and the agent would start and then would produce the certificate files that cloud-init needed (for ssh keys and things). I found this when testing SRU for 0.7.7-31-g65ace7b-0ubuntu1~16.04.1 but it is likely present also in 0.7.7-31-g65ace7b-0ubuntu1 (yakkety) Now, however we see a log like: Sep 14 14:53:18 smoser0914x [CLOUDINIT] DataSourceAzure.py[DEBUG]: Getting metadata via agent. hostname=smoser0914x cmd=['service', 'walinuxagent', 'start'] Sep 14 14:53:18 smoser0914x [CLOUDINIT] util.py[DEBUG]: Running command hostname with allowed return codes [0] (shell=False, capture=True) Sep 14 14:53:18 smoser0914x [CLOUDINIT] DataSourceAzure.py[DEBUG]: invoking agent: ['service', 'walinuxagent', 'start'] Sep 14 14:53:18 smoser0914x [CLOUDINIT] util.py[DEBUG]: Running command ['service', 'walinuxagent', 'start'] with allowed return codes [0] (shell=False, capture=True) Sep 14 14:53:18 smoser0914x [CLOUDINIT] util.py[WARNING]: agent command '['service', 'walinuxagent', 'start']' failed. Sep 14 14:53:19 smoser0914x [CLOUDINIT] util.py[DEBUG]: agent command '['service', 'walinuxagent', 'start']' failed. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceAzure.py", line 145, in get_metadata_from_agent invoke_agent(agent_cmd) File "/usr/lib/python3/dist-packages/cloudinit/sources/DataSourceAzure.py", line 452, in invoke_agent util.subp(cmd, shell=(not isinstance(cmd, list))) File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1832, in subp cmd=args) cloudinit.util.ProcessExecutionError: Unexpected error while running command. Command: ['service', 'walinuxagent', 'start'] Exit code: 1 Reason: - Stdout: '' Stderr: " Failed to start walinuxagent.service: Transaction order is cyclic. See system logs for details. See system logs and 'systemctl status walinuxagent.service' for details I believe the relevant change is in 34a26f7f https://git.launchpad.net/cloud-init/commit/?id=34a26f7f59f2963691e36ca0476bec9fc9ccef63 That added multi-user.target to the list of After for cloud-init-final.service. Related bugs: * bug 1576692: fully support package installation in systemd To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/walinuxagent/+bug/1623570/+subscriptions _______________________________________________ Mailing list: https://launchpad.net/~group.of.nepali.translators Post to : group.of.nepali.translators@lists.launchpad.net Unsubscribe : https://launchpad.net/~group.of.nepali.translators More help : https://help.launchpad.net/ListHelp