Public bug reported: When the juju model config includes a `snap-store-assertions`, the cloud-init.service runs into problem when trying to contact snapd's socket.
In this setup, cloud-ctrl01 runs a Juju controller in a LXD container and cloud-vm02 is a VM created by MAAS. Here is how to reproduce the issue: ubuntu@cloud-ctrl01:~$ juju model-config ... snap-store-assertions model |- type: account-key authority-id: canonical revision: 2 ... snap-store-proxy model dI5E5ZV6U3wOc919eLmZ0MtOxAyxxTIP snap-store-proxy-url default "" ... ubuntu@cloud-ctrl01:~$ juju add-machine created machine 2 ubuntu@cloud-ctrl01:~$ juju ssh 2 ... Inside "machine 2", cloud-init's journal output: ubuntu@cloud-vm02:~$ journalctl -u cloud-init.service | cat -- Logs begin at Wed 2021-07-28 21:06:26 UTC, end at Wed 2021-07-28 21:17:01 UTC. -- Jul 28 21:06:31 ubuntu systemd[1]: Starting Initial cloud-init job (metadata service crawler)... Jul 28 21:06:33 cloud-vm02 cloud-init[893]: Cloud-init v. 21.2-3-g899bfaa9-0ubuntu2~20.04.1 running 'init' at Wed, 28 Jul 2021 21:06:32 +0000. Up 10.23 seconds. Jul 28 21:06:33 cloud-vm02 cloud-init[893]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++ Jul 28 21:06:33 cloud-vm02 cloud-init[893]: ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+ Jul 28 21:06:33 cloud-vm02 cloud-init[893]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | ... Jul 28 21:06:33 cloud-vm02 cloud-init[893]: ci-info: +-------+-----------------------+---------------------+-----------+-------+ Jul 28 21:06:33 cloud-vm02 cloud-init[893]: error: cannot assert: cannot communicate with server: Post http://localhost/v2/assertions: dial unix /run/snapd.socket: connect: no such file or directory Jul 28 21:06:33 cloud-vm02 cloud-init[893]: error: cannot communicate with server: Put http://localhost/v2/snaps/core/conf: dial unix /run/snapd.socket: connect: no such file or directory Jul 28 21:06:33 cloud-vm02 cloud-init[893]: 2021-07-28 21:06:33,810 - util.py[WARNING]: Failed to run bootcmd module bootcmd Jul 28 21:06:33 cloud-vm02 cloud-init[893]: 2021-07-28 21:06:33,822 - util.py[WARNING]: Running module bootcmd (<module 'cloudinit.config.cc_bootcmd' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_bootcmd.py'>) failed Jul 28 21:06:34 cloud-vm02 useradd[989]: new group: name=ubuntu, GID=1000 ... Jul 28 21:06:35 cloud-vm02 systemd[1]: cloud-init.service: Main process exited, code=exited, status=1/FAILURE Jul 28 21:06:35 cloud-vm02 systemd[1]: cloud-init.service: Failed with result 'exit-code'. Jul 28 21:06:35 cloud-vm02 systemd[1]: Failed to start Initial cloud-init job (metadata service crawler). Removing the snap-store-assertions/snap-store-proxy configs from the model make cloud-init work again. FYI, my naive attempt at adding "After=sockets.target" to the cloud- init.service didn't work :/ Additional information: ubuntu@cloud-vm02:~$ /var/lib/juju/tools/machine-2/jujud version 2.9.9-ubuntu-amd64 $ lsb_release -rd Description: Ubuntu 20.04.2 LTS Release: 20.04 ubuntu@cloud-vm02:~$ apt-cache policy cloud-init cloud-init: Installed: 21.2-3-g899bfaa9-0ubuntu2~20.04.1 Candidate: 21.2-3-g899bfaa9-0ubuntu2~20.04.1 Version table: *** 21.2-3-g899bfaa9-0ubuntu2~20.04.1 500 500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages 100 /var/lib/dpkg/status 20.1-10-g71af48df-0ubuntu5 500 500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages ubuntu@cloud-ctrl01:~$ juju version 2.9.9-ubuntu-amd64 ** Affects: cloud-init Importance: Undecided Status: New ** Affects: juju Importance: Undecided Status: New ** Also affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1938344 Title: Setting snap-store-assertions in a model config causes cloud-init to fail Status in cloud-init: New Status in juju: New Bug description: When the juju model config includes a `snap-store-assertions`, the cloud-init.service runs into problem when trying to contact snapd's socket. In this setup, cloud-ctrl01 runs a Juju controller in a LXD container and cloud-vm02 is a VM created by MAAS. Here is how to reproduce the issue: ubuntu@cloud-ctrl01:~$ juju model-config ... snap-store-assertions model |- type: account-key authority-id: canonical revision: 2 ... snap-store-proxy model dI5E5ZV6U3wOc919eLmZ0MtOxAyxxTIP snap-store-proxy-url default "" ... ubuntu@cloud-ctrl01:~$ juju add-machine created machine 2 ubuntu@cloud-ctrl01:~$ juju ssh 2 ... Inside "machine 2", cloud-init's journal output: ubuntu@cloud-vm02:~$ journalctl -u cloud-init.service | cat -- Logs begin at Wed 2021-07-28 21:06:26 UTC, end at Wed 2021-07-28 21:17:01 UTC. -- Jul 28 21:06:31 ubuntu systemd[1]: Starting Initial cloud-init job (metadata service crawler)... Jul 28 21:06:33 cloud-vm02 cloud-init[893]: Cloud-init v. 21.2-3-g899bfaa9-0ubuntu2~20.04.1 running 'init' at Wed, 28 Jul 2021 21:06:32 +0000. Up 10.23 seconds. Jul 28 21:06:33 cloud-vm02 cloud-init[893]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++ Jul 28 21:06:33 cloud-vm02 cloud-init[893]: ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+ Jul 28 21:06:33 cloud-vm02 cloud-init[893]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address | ... Jul 28 21:06:33 cloud-vm02 cloud-init[893]: ci-info: +-------+-----------------------+---------------------+-----------+-------+ Jul 28 21:06:33 cloud-vm02 cloud-init[893]: error: cannot assert: cannot communicate with server: Post http://localhost/v2/assertions: dial unix /run/snapd.socket: connect: no such file or directory Jul 28 21:06:33 cloud-vm02 cloud-init[893]: error: cannot communicate with server: Put http://localhost/v2/snaps/core/conf: dial unix /run/snapd.socket: connect: no such file or directory Jul 28 21:06:33 cloud-vm02 cloud-init[893]: 2021-07-28 21:06:33,810 - util.py[WARNING]: Failed to run bootcmd module bootcmd Jul 28 21:06:33 cloud-vm02 cloud-init[893]: 2021-07-28 21:06:33,822 - util.py[WARNING]: Running module bootcmd (<module 'cloudinit.config.cc_bootcmd' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_bootcmd.py'>) failed Jul 28 21:06:34 cloud-vm02 useradd[989]: new group: name=ubuntu, GID=1000 ... Jul 28 21:06:35 cloud-vm02 systemd[1]: cloud-init.service: Main process exited, code=exited, status=1/FAILURE Jul 28 21:06:35 cloud-vm02 systemd[1]: cloud-init.service: Failed with result 'exit-code'. Jul 28 21:06:35 cloud-vm02 systemd[1]: Failed to start Initial cloud-init job (metadata service crawler). Removing the snap-store-assertions/snap-store-proxy configs from the model make cloud-init work again. FYI, my naive attempt at adding "After=sockets.target" to the cloud- init.service didn't work :/ Additional information: ubuntu@cloud-vm02:~$ /var/lib/juju/tools/machine-2/jujud version 2.9.9-ubuntu-amd64 $ lsb_release -rd Description: Ubuntu 20.04.2 LTS Release: 20.04 ubuntu@cloud-vm02:~$ apt-cache policy cloud-init cloud-init: Installed: 21.2-3-g899bfaa9-0ubuntu2~20.04.1 Candidate: 21.2-3-g899bfaa9-0ubuntu2~20.04.1 Version table: *** 21.2-3-g899bfaa9-0ubuntu2~20.04.1 500 500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages 100 /var/lib/dpkg/status 20.1-10-g71af48df-0ubuntu5 500 500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages ubuntu@cloud-ctrl01:~$ juju version 2.9.9-ubuntu-amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1938344/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
