** Description changed: - From time to time (sporadically and very rarely - maybe in one or two - attempt out of twenty) I face a situation where the installer system (on - s390x) does not boot-up completely. + [impact] + When there is a serial console configured, there was a unit cycle + + serial-getty@$TTY.service -> cloud-final.service -> multi-user.target -> + getty.target -> serial-getty@$TTY.service + + (or something like that) + + Depending on which unit systemd kills to resolve the cycle, this can + result in cloud-init never completing which leads to the subiquity + server waiting forever for it and nothing useful can be done (other than + restarting and hoping for better luck next time). + + Because subiquity itself waits for cloud-init (and this has been true + for a long time now) there is no need for serial-getty@$TTY.service to + wait on cloud-final.service. + + [regression potential] + This change results in shuffling the systemd units around a fair bit, but the new arrangement has been tested in devel for a few months now and works well there. It's also much more straightforward than the current setup. + + [test case] + This is a bit tricky as it's an intermittent failure. Basically, boot the live installer with a serial console configured a bunch of times and (10?) check that the installer starts up properly each time. + + [original description] + From time to time (sporadically and very rarely - maybe in one or two attempt out of twenty) I face a situation where the installer system (on s390x) does not boot-up completely. This happened to me in the past already, but since it happened only one or twice I thought it's due to resource constraints on the system or so. But since I faced it now again on LPAR (before it was on z/VM), I'm opening this ticket now. In the latest case I used the focal daily live image from the 20th of July with installer 20.06.1 (but that also happened with previous versions). The situation is like this: In boot of the installer ends here (LPAR): ... "[ 128.200711] cloud-init[1375]: The key's randomart image is:" "[ 128.200735] cloud-init[1375]: +--[ED25519 256]--+" "[ 128.200758] cloud-init[1375]: |o .....ooo |" "[ 128.200781] cloud-init[1375]: |.= . . +. o. .|" "[ 128.200804] cloud-init[1375]: |+ * . . * o.o . |" "[ 128.200826] cloud-init[1375]: |.= o . = = = + |" "[ 128.200849] cloud-init[1375]: |o + . S o + |" "[ 128.200876] cloud-init[1375]: | + o = . . |" "[ 128.200900] cloud-init[1375]: | o + . |" "[ 128.200925] cloud-init[1375]: | o.=. E |" "[ 128.200947] cloud-init[1375]: | .+.o+o. |" "[ 128.200977] cloud-init[1375]: +----[SHA256]-----+" "[ 138.898906] cloud-init[2217]: Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~20.04." "1 running 'modules:config' at Wed, 22 Jul 2020 11:27:39 +0000. Up 138.77 seconds" . "[ 138.898966] cloud-init[2217]: Set the following 'random' passwords" "[ 138.899001] cloud-init[2217]: installer:aecmewaoicnai" or another example (z/VM): ... - ¬ 93.463680| cloud-init¬1282|: +--¬ED25519 256|--+ - ¬ 93.463713| cloud-init¬1282|: !Eo=o .... ! - ¬ 93.463749| cloud-init¬1282|: !.Bo.o ... o ! - ¬ 93.463782| cloud-init¬1282|: !**.*... o = ! - ¬ 93.463818| cloud-init¬1282|: !*=O o. o . . ! - ¬ 93.463849| cloud-init¬1282|: !**++ S ! - ¬ 93.463886| cloud-init¬1282|: !§o+.. ! - ¬ 93.463918| cloud-init¬1282|: !+*o. ! - ¬ 93.463954| cloud-init¬1282|: !.o. ! - ¬ 93.463988| cloud-init¬1282|: !. ! - ¬ 93.464028| cloud-init¬1282|: +----¬SHA256|-----+ + ¬ 93.463680| cloud-init¬1282|: +--¬ED25519 256|--+ + ¬ 93.463713| cloud-init¬1282|: !Eo=o .... ! + ¬ 93.463749| cloud-init¬1282|: !.Bo.o ... o ! + ¬ 93.463782| cloud-init¬1282|: !**.*... o = ! + ¬ 93.463818| cloud-init¬1282|: !*=O o. o . . ! + ¬ 93.463849| cloud-init¬1282|: !**++ S ! + ¬ 93.463886| cloud-init¬1282|: !§o+.. ! + ¬ 93.463918| cloud-init¬1282|: !+*o. ! + ¬ 93.463954| cloud-init¬1282|: !.o. ! + ¬ 93.463988| cloud-init¬1282|: !. ! + ¬ 93.464028| cloud-init¬1282|: +----¬SHA256|-----+ ¬ 104.841438| cloud-init¬2004|: Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1ß20.04. 1 running 'modules:config' at Mon, 20 Jul 2020 10:46:38 +0000. Up 104.63 seconds - . - ¬ 104.841490| cloud-init¬2004|: Set the following 'random' passwords - ¬ 104.841516| cloud-init¬2004|: installer:U9NJDuvXFw6X2fxG7pP8 + . + ¬ 104.841490| cloud-init¬2004|: Set the following 'random' passwords + ¬ 104.841516| cloud-init¬2004|: installer:U9NJDuvXFw6X2fxG7pP8 But it is not complete at this point. A completed boot of the installer system ends like this: "It is possible to connect to the installer over the network, which" "might allow the use of a more capable terminal and can offer more languages" "than can be rendered in the Linux console." - "To connect, SSH to installer@<IP address>." - "The password you should use is "ydnjdnciu"kZ4tR4vRvPxHPerCNU8g""." "The host key fingerprints are:" "RSA SHA256:n+6TJsfdCBII2PO89GMU10mG1oFvEIFBT2v6uPN0Jz0" "ECDSA SHA256:VcDS5ac8xswXxFEghjo1ZIcue38AM6HJg0poIxdeeec" "ED25519 SHA256:lb/DVVhj1obDPhfo3M8oPqeAyduvlLcPFJCC8ZaiCJY" "Ubuntu 20.04 LTS ubuntu-server sclp_line0" - In such an above situation I also can't reach the subiquity UI: fheimes@T570:~$ ssh-keygen -f "/home/fheimes/.ssh/known_hosts" -R s1lp14 # Host s1lp14 found: line 165 /home/fheimes/.ssh/known_hosts updated. Original contents retained as /home/fheimes/.ssh/known_hosts.old fheimes@T570:~$ ssh installer@s1lp14 The authenticity of host 's1lp14 (10.245.236.14)' can't be established. ECDSA key fingerprint is SHA256:AoYrJhWOFaaEtUun/aZAT8/VgNP+yq//Jhg7vzUwvxE. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 's1lp14,10.245.236.14' (ECDSA) to the list of known hosts. - installer@s1lp14's password: + installer@s1lp14's password: Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-40-generic s390x) - * Documentation: https://help.ubuntu.com - * Management: https://landscape.canonical.com - * Support: https://ubuntu.com/advantage + * Documentation: https://help.ubuntu.com + * Management: https://landscape.canonical.com + * Support: https://ubuntu.com/advantage - System information as of Wed Jul 22 11:28:32 UTC 2020 + System information as of Wed Jul 22 11:28:32 UTC 2020 - System load: 0.44 Memory usage: 4% Processes: 180 - Usage of /home: unknown Swap usage: 0% Users logged in: 0 + System load: 0.44 Memory usage: 4% Processes: 180 + Usage of /home: unknown Swap usage: 0% Users logged in: 0 0 updates can be installed immediately. 0 of these updates are security updates. - The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. - Hence even gathering the logs is unfortunately not easily possible.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1888497 Title: Booting the install system does not always succeed, hence a remote ssh login is not always possible To manage notifications about this bug go to: https://bugs.launchpad.net/subiquity/+bug/1888497/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
