** Description changed: - With current yakkety cloud images (at least in Scalingstack), I often - run into this dependency cycle at boot: + ==== Begin SRU Template ==== + [Impact] + As part of the change in bug 1623868, we made cloud-final.target + run After multi-user.target. That created a dependency loop between + cloud-init.target and multi-user.target and cloud-final.target. + + Most of the time systemd would break that loop by dropping + cloud-init.target. But sometimes, it would break the loop by dropping + cloud-final.target, which would mean that user scripts do not run + and generally cloud-init doesn't finish. + + [Test Case] + ## Failure in a xenial image can only be reproduced by + ## patching an image up to the previous xenial-proposed upload + ## (0.7.7-31-g65ace7b-0ubuntu1~16.04.1), then cleaning it + ## and then restarting. We will focus on verifying there is not + ## a problem. + + ## Launch an instance and patch it up to xenial-proposed + $ release=xenial + $ name=x1 + $ lxc launch ubuntu-daily:$release $name + + # wait for it to boot + $ while ! lxc exec $name -- [ -e /run/cloud-init/result.json ]; do sleep 1; done + + ## Now update container, clean and reboot to show first boot + $ lxc exec $name -- sh -c ' + p=/etc/apt/sources.list.d/proposed.list + echo deb http://archive.ubuntu.com/ubuntu xenial-proposed main > "$p" && + apt-get update -q && apt-get -qy install cloud-init' + $ lxc exec $name -- sh -c ' + cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done + rm -Rf /var/log/cloud-init*' + + ## This is like first boot now. + $ lxc exec $name reboot + $ while ! lxc exec $name -- [ -e /run/cloud-init/result.json ]; do sleep 1; done + + ## The services should show active + $ lxc exec $name -- journalctl | grep Break || echo Good, no breaks + $ lxc exec $name -- systemctl --no-pager status cloud-final.service + $ lxc exec $name -- systemctl --no-pager status cloud-init.target + + [Regression Potential] + Playing with boot order can cause problems. Regression would be around + some targets not running. On a booted system this would show itself inx + journalctl | grep -i Break + or + journalctl | grep -i ordering + + + ==== End SRU Template ==== + + + With current yakkety cloud images (at least in Scalingstack), I often run into this dependency cycle at boot: Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found ordering cycle on cloud-init.target/start Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-final.service/start Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on multi-user.target/start Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-init.target/start Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Breaking ordering cycle by deleting job cloud-final.service/start Sep 15 09:28:51 ubuntu systemd[1]: cloud-final.service: Job cloud-final.service/start deleted to break ordering cycle starting with cloud-init.target/start ● cloud-final.service - Execute cloud user/final scripts - Loaded: loaded (/lib/systemd/system/cloud-final.service; enabled; vendor preset: enabled) - Active: inactive (dead) + Loaded: loaded (/lib/systemd/system/cloud-final.service; enabled; vendor preset: enabled) + Active: inactive (dead) - - Thus /var/lib/cloud/instance/boot-finished never gets written and thus waiting for an instance to init just times out. + Thus /var/lib/cloud/instance/boot-finished never gets written and thus + waiting for an instance to init just times out. This is with the most recent https://launchpad.net/ubuntu/+source/cloud- init/0.7.7-31-g65ace7b-0ubuntu1
** Description changed: ==== Begin SRU Template ==== - [Impact] - As part of the change in bug 1623868, we made cloud-final.target + [Impact] + As part of the change in bug 1576692, we made cloud-final.target run After multi-user.target. That created a dependency loop between cloud-init.target and multi-user.target and cloud-final.target. - Most of the time systemd would break that loop by dropping + Most of the time systemd would break that loop by dropping cloud-init.target. But sometimes, it would break the loop by dropping cloud-final.target, which would mean that user scripts do not run and generally cloud-init doesn't finish. [Test Case] ## Failure in a xenial image can only be reproduced by ## patching an image up to the previous xenial-proposed upload ## (0.7.7-31-g65ace7b-0ubuntu1~16.04.1), then cleaning it ## and then restarting. We will focus on verifying there is not ## a problem. ## Launch an instance and patch it up to xenial-proposed $ release=xenial $ name=x1 $ lxc launch ubuntu-daily:$release $name # wait for it to boot $ while ! lxc exec $name -- [ -e /run/cloud-init/result.json ]; do sleep 1; done - ## Now update container, clean and reboot to show first boot + ## Now update container, clean and reboot to show first boot $ lxc exec $name -- sh -c ' - p=/etc/apt/sources.list.d/proposed.list - echo deb http://archive.ubuntu.com/ubuntu xenial-proposed main > "$p" && - apt-get update -q && apt-get -qy install cloud-init' + p=/etc/apt/sources.list.d/proposed.list + echo deb http://archive.ubuntu.com/ubuntu xenial-proposed main > "$p" && + apt-get update -q && apt-get -qy install cloud-init' $ lxc exec $name -- sh -c ' - cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done - rm -Rf /var/log/cloud-init*' + cd /var/lib/cloud && for d in *; do [ "$d" = "seed" ] || rm -Rf "$d"; done + rm -Rf /var/log/cloud-init*' ## This is like first boot now. $ lxc exec $name reboot $ while ! lxc exec $name -- [ -e /run/cloud-init/result.json ]; do sleep 1; done ## The services should show active $ lxc exec $name -- journalctl | grep Break || echo Good, no breaks $ lxc exec $name -- systemctl --no-pager status cloud-final.service $ lxc exec $name -- systemctl --no-pager status cloud-init.target - [Regression Potential] + [Regression Potential] Playing with boot order can cause problems. Regression would be around some targets not running. On a booted system this would show itself inx - journalctl | grep -i Break + journalctl | grep -i Break or - journalctl | grep -i ordering - + journalctl | grep -i ordering ==== End SRU Template ==== - - With current yakkety cloud images (at least in Scalingstack), I often run into this dependency cycle at boot: + With current yakkety cloud images (at least in Scalingstack), I often + run into this dependency cycle at boot: Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found ordering cycle on cloud-init.target/start Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-final.service/start Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on multi-user.target/start Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Found dependency on cloud-init.target/start Sep 15 09:28:51 ubuntu systemd[1]: cloud-init.target: Breaking ordering cycle by deleting job cloud-final.service/start Sep 15 09:28:51 ubuntu systemd[1]: cloud-final.service: Job cloud-final.service/start deleted to break ordering cycle starting with cloud-init.target/start ● cloud-final.service - Execute cloud user/final scripts Loaded: loaded (/lib/systemd/system/cloud-final.service; enabled; vendor preset: enabled) Active: inactive (dead) Thus /var/lib/cloud/instance/boot-finished never gets written and thus waiting for an instance to init just times out. This is with the most recent https://launchpad.net/ubuntu/+source/cloud- init/0.7.7-31-g65ace7b-0ubuntu1 + + Related bugs: + * bug 1576692: fully support package installation in systemd -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1623868 Title: cloud-final.service does not run due to dependency cycle To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1623868/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
