[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
This bug was fixed in the package console-setup - 1.108ubuntu15.5 --- console-setup (1.108ubuntu15.5) xenial; urgency=medium * setupcon: use /run for tempfiles (and dump the various unnecessary fallback paths), since /run is always mountable rw at least as early as /tmp is and is guaranteed to be safe from tmpcleaners at boot. Only keep /tmp as a fallback in case we have access to write to /tmp and to a console, but not to /run. LP: #1824227. -- Steve Langasek Wed, 10 Apr 2019 13:16:29 -0700 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Fix Released Status in console-setup source package in Xenial: Fix Released Status in console-setup source package in Bionic: Fix Released Status in console-setup source package in Cosmic: Fix Released Status in console-setup source package in Disco: Fix Released Bug description: [SRU Justification] I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot [Test case] 1. Install console-setup from -proposed 2. Reboot 3. Verify that `systemctl status console-setup` shows that the service has completed successfully. Since this is a race, additional fuzz testing may be warranted for the cloud images to confirm that the issue experienced in GCE is really fixed. However, that should not block promotion of this SRU fix since there definitely is a race here that should be fixed per se even if there are other issues still causing a failure in GCE. [Regression potential] None known. /run is guaranteed to be mounted rw very early in boot - generally before /tmp is mounted, due to /tmp being on the rootfs that needs to be fscked before remount. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: console-setup UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to:
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
This bug was fixed in the package console-setup - 1.178ubuntu2.8 --- console-setup (1.178ubuntu2.8) bionic; urgency=medium * setupcon: use /run for tempfiles (and dump the various unnecessary fallback paths), since /run is always mountable rw at least as early as /tmp is and is guaranteed to be safe from tmpcleaners at boot. Only keep /tmp as a fallback in case we have access to write to /tmp and to a console, but not to /run. LP: #1824227. -- Steve Langasek Wed, 10 Apr 2019 13:13:34 -0700 ** Changed in: console-setup (Ubuntu Bionic) Status: Fix Committed => Fix Released ** Changed in: console-setup (Ubuntu Xenial) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Fix Released Status in console-setup source package in Xenial: Fix Released Status in console-setup source package in Bionic: Fix Released Status in console-setup source package in Cosmic: Fix Released Status in console-setup source package in Disco: Fix Released Bug description: [SRU Justification] I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot [Test case] 1. Install console-setup from -proposed 2. Reboot 3. Verify that `systemctl status console-setup` shows that the service has completed successfully. Since this is a race, additional fuzz testing may be warranted for the cloud images to confirm that the issue experienced in GCE is really fixed. However, that should not block promotion of this SRU fix since there definitely is a race here that should be fixed per se even if there are other issues still causing a failure in GCE. [Regression potential] None known. /run is guaranteed to be mounted rw very early in boot - generally before /tmp is mounted, due to /tmp being on the rootfs that needs to be fscked before remount. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: console-setup
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
This bug was fixed in the package console-setup - 1.178ubuntu9.1 --- console-setup (1.178ubuntu9.1) cosmic; urgency=medium * setupcon: use /run for tempfiles (and dump the various unnecessary fallback paths), since /run is always mountable rw at least as early as /tmp is and is guaranteed to be safe from tmpcleaners at boot. Only keep /tmp as a fallback in case we have access to write to /tmp and to a console, but not to /run. LP: #1824227. -- Steve Langasek Wed, 10 Apr 2019 13:02:51 -0700 ** Changed in: console-setup (Ubuntu Cosmic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Fix Released Status in console-setup source package in Xenial: Fix Released Status in console-setup source package in Bionic: Fix Released Status in console-setup source package in Cosmic: Fix Released Status in console-setup source package in Disco: Fix Released Bug description: [SRU Justification] I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot [Test case] 1. Install console-setup from -proposed 2. Reboot 3. Verify that `systemctl status console-setup` shows that the service has completed successfully. Since this is a race, additional fuzz testing may be warranted for the cloud images to confirm that the issue experienced in GCE is really fixed. However, that should not block promotion of this SRU fix since there definitely is a race here that should be fixed per se even if there are other issues still causing a failure in GCE. [Regression potential] None known. /run is guaranteed to be mounted rw very early in boot - generally before /tmp is mounted, due to /tmp being on the rootfs that needs to be fscked before remount. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: console-setup UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
I have built cosmic, bionic, and xenial images in GCE with the packages from -proposed and tested that console setup is functioning correctly with the change. Thanks. I will be marking each of the releases as tested. ** Tags removed: verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-xenial ** Tags added: verification-done verification-done-bionic verification-done-cosmic verification-done-xenial -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Fix Released Status in console-setup source package in Xenial: Fix Committed Status in console-setup source package in Bionic: Fix Committed Status in console-setup source package in Cosmic: Fix Committed Status in console-setup source package in Disco: Fix Released Bug description: [SRU Justification] I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot [Test case] 1. Install console-setup from -proposed 2. Reboot 3. Verify that `systemctl status console-setup` shows that the service has completed successfully. Since this is a race, additional fuzz testing may be warranted for the cloud images to confirm that the issue experienced in GCE is really fixed. However, that should not block promotion of this SRU fix since there definitely is a race here that should be fixed per se even if there are other issues still causing a failure in GCE. [Regression potential] None known. /run is guaranteed to be mounted rw very early in boot - generally before /tmp is mounted, due to /tmp being on the rootfs that needs to be fscked before remount. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: console-setup UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/console-setup/+bug/1824227/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to :
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
** Tags added: id-5cae47869ef68677a65c1e71 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Fix Released Status in console-setup source package in Xenial: Fix Committed Status in console-setup source package in Bionic: Fix Committed Status in console-setup source package in Cosmic: Fix Committed Status in console-setup source package in Disco: Fix Released Bug description: [SRU Justification] I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot [Test case] 1. Install console-setup from -proposed 2. Reboot 3. Verify that `systemctl status console-setup` shows that the service has completed successfully. Since this is a race, additional fuzz testing may be warranted for the cloud images to confirm that the issue experienced in GCE is really fixed. However, that should not block promotion of this SRU fix since there definitely is a race here that should be fixed per se even if there are other issues still causing a failure in GCE. [Regression potential] None known. /run is guaranteed to be mounted rw very early in boot - generally before /tmp is mounted, due to /tmp being on the rootfs that needs to be fscked before remount. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: console-setup UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/console-setup/+bug/1824227/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
** Tags added: id-5caba54405de185b64adc2db -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Fix Released Status in console-setup source package in Xenial: Fix Committed Status in console-setup source package in Bionic: Fix Committed Status in console-setup source package in Cosmic: Fix Committed Status in console-setup source package in Disco: Fix Released Bug description: [SRU Justification] I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot [Test case] 1. Install console-setup from -proposed 2. Reboot 3. Verify that `systemctl status console-setup` shows that the service has completed successfully. Since this is a race, additional fuzz testing may be warranted for the cloud images to confirm that the issue experienced in GCE is really fixed. However, that should not block promotion of this SRU fix since there definitely is a race here that should be fixed per se even if there are other issues still causing a failure in GCE. [Regression potential] None known. /run is guaranteed to be mounted rw very early in boot - generally before /tmp is mounted, due to /tmp being on the rootfs that needs to be fscked before remount. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: console-setup UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/console-setup/+bug/1824227/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
This bug was fixed in the package console-setup - 1.178ubuntu12 --- console-setup (1.178ubuntu12) disco; urgency=medium * setupcon: use /run for tempfiles (and dump the various unnecessary fallback paths), since /run is always mountable rw at least as early as /tmp is and is guaranteed to be safe from tmpcleaners at boot. Only keep /tmp as a fallback in case we have access to write to /tmp and to a console, but not to /run. LP: #1824227. -- Steve Langasek Wed, 10 Apr 2019 12:51:56 -0700 ** Changed in: console-setup (Ubuntu Disco) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Fix Released Status in console-setup source package in Xenial: Fix Committed Status in console-setup source package in Bionic: Fix Committed Status in console-setup source package in Cosmic: Fix Committed Status in console-setup source package in Disco: Fix Released Bug description: [SRU Justification] I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot [Test case] 1. Install console-setup from -proposed 2. Reboot 3. Verify that `systemctl status console-setup` shows that the service has completed successfully. Since this is a race, additional fuzz testing may be warranted for the cloud images to confirm that the issue experienced in GCE is really fixed. However, that should not block promotion of this SRU fix since there definitely is a race here that should be fixed per se even if there are other issues still causing a failure in GCE. [Regression potential] None known. /run is guaranteed to be mounted rw very early in boot - generally before /tmp is mounted, due to /tmp being on the rootfs that needs to be fscked before remount. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: console-setup UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
Hello Robert, or anyone else affected, Accepted console-setup into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/console- setup/1.178ubuntu9.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: console-setup (Ubuntu Cosmic) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-cosmic ** Changed in: console-setup (Ubuntu Bionic) Status: In Progress => Fix Committed ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Fix Committed Status in console-setup source package in Xenial: Fix Committed Status in console-setup source package in Bionic: Fix Committed Status in console-setup source package in Cosmic: Fix Committed Status in console-setup source package in Disco: Fix Committed Bug description: [SRU Justification] I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot [Test case] 1. Install console-setup from -proposed 2. Reboot 3. Verify that `systemctl status console-setup` shows that the service has completed successfully. Since this is a race, additional fuzz testing may be warranted for the cloud images to confirm that the issue experienced in GCE is really fixed. However, that should not block promotion of this SRU fix since there
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
** Changed in: console-setup (Ubuntu Cosmic) Status: Triaged => In Progress ** Changed in: console-setup (Ubuntu Bionic) Status: Triaged => In Progress ** Changed in: console-setup (Ubuntu Xenial) Status: New => In Progress ** Changed in: console-setup (Ubuntu Cosmic) Assignee: (unassigned) => Steve Langasek (vorlon) ** Changed in: console-setup (Ubuntu Bionic) Assignee: (unassigned) => Steve Langasek (vorlon) ** Changed in: console-setup (Ubuntu Xenial) Assignee: (unassigned) => Steve Langasek (vorlon) ** Changed in: console-setup (Ubuntu Xenial) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Fix Committed Status in console-setup source package in Xenial: In Progress Status in console-setup source package in Bionic: In Progress Status in console-setup source package in Cosmic: In Progress Status in console-setup source package in Disco: Fix Committed Bug description: [SRU Justification] I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot [Test case] 1. Install console-setup from -proposed 2. Reboot 3. Verify that `systemctl status console-setup` shows that the service has completed successfully. Since this is a race, additional fuzz testing may be warranted for the cloud images to confirm that the issue experienced in GCE is really fixed. However, that should not block promotion of this SRU fix since there definitely is a race here that should be fixed per se even if there are other issues still causing a failure in GCE. [Regression potential] None known. /run is guaranteed to be mounted rw very early in boot - generally before /tmp is mounted, due to /tmp being on the rootfs that needs to be fscked before remount. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: console-setup UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
** Description changed: + [SRU Justification] I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. - 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. - ... - 19:51:13 systemd[1]: Starting Set console font and keymap... - 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file - 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE - 19:51:15 systemd[1]: Failed to start Set console font and keymap. - 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. - 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. - ... + 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. + ... + 19:51:13 systemd[1]: Starting Set console font and keymap... + 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file + 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE + 19:51:15 systemd[1]: Failed to start Set console font and keymap. + 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. + 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. + ... /bin/setupcon has a lovely tempfile function that looks like: - if \ - TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ - || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ - || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ - || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ - || TMPFILE=`mktemp 2>/dev/null` + if \ + TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ + || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ + || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ + || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ + || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for - the tmp cleaner, to delete files that have just been created + the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with - the ordering of the systemd units on startup + the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency - on systemd-tmpfiles-clean; let's remove the dependency - instead of declaring it + on systemd-tmpfiles-clean; let's remove the dependency + instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have - data to answer that. + data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, - say "yep, nothing to do here" and exit + say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow - to complete AIUI + to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because - that will happen + that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot + + [Test case] + 1. Install console-setup from -proposed + 2. Reboot + 3. Verify that `systemctl status console-setup` shows that the service has completed successfully. + + Since this is a race, additional fuzz testing may be warranted for the + cloud images to confirm that the issue experienced in GCE is really + fixed. However, that should not block promotion of this SRU fix since + there definitely is a race here that should be fixed per se even if + there are other issues still causing a failure in GCE. + + [Regression potential] + None known. /run is guaranteed to be mounted rw very early in boot - generally before /tmp is mounted, due to /tmp being on the rootfs that needs to be fscked before remount. + ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: - TERM=xterm-256color - PATH=(custom, no user) - XDG_RUNTIME_DIR= - LANG=en_US.UTF-8 -
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
** Changed in: console-setup (Ubuntu Disco) Status: Triaged => Fix Committed ** Changed in: console-setup (Ubuntu Disco) Assignee: (unassigned) => Steve Langasek (vorlon) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Fix Committed Status in console-setup source package in Bionic: Triaged Status in console-setup source package in Cosmic: Triaged Status in console-setup source package in Disco: Fix Committed Bug description: I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: console-setup UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/console-setup/+bug/1824227/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
** Changed in: console-setup (Ubuntu) Importance: Undecided => High ** Changed in: console-setup (Ubuntu) Status: New => Triaged ** Also affects: console-setup (Ubuntu Disco) Importance: High Status: Triaged ** Also affects: console-setup (Ubuntu Bionic) Importance: Undecided Status: New ** Also affects: console-setup (Ubuntu Cosmic) Importance: Undecided Status: New ** Changed in: console-setup (Ubuntu Bionic) Status: New => Triaged ** Changed in: console-setup (Ubuntu Bionic) Importance: Undecided => High ** Changed in: console-setup (Ubuntu Cosmic) Status: New => Triaged ** Changed in: console-setup (Ubuntu Cosmic) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Triaged Status in console-setup source package in Bionic: Triaged Status in console-setup source package in Cosmic: Triaged Status in console-setup source package in Disco: Triaged Bug description: I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: console-setup UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/console-setup/+bug/1824227/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1824227] Re: console-setup failure due to race with systemd-tmpfiles
** Attachment added: "journal" https://bugs.launchpad.net/ubuntu/+source/console-setup/+bug/1824227/+attachment/5254776/+files/journal -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to console-setup in Ubuntu. https://bugs.launchpad.net/bugs/1824227 Title: console-setup failure due to race with systemd-tmpfiles Status in console-setup package in Ubuntu: Triaged Status in console-setup source package in Bionic: Triaged Status in console-setup source package in Cosmic: Triaged Status in console-setup source package in Disco: Triaged Bug description: I'm seeing a console-setup.service failure quite regularly in testing where the temp file that should have been created can't be found. This is a regular xenial cloud image. 19:51:13 systemd-tmpfiles[485]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring. ... 19:51:13 systemd[1]: Starting Set console font and keymap... 19:51:15 setupcon[455]: /bin/setupcon: 809: /bin/setupcon: cannot open /tmp/tmpkbd.a8FGSs: No such file 19:51:15 systemd[1]: console-setup.service: Main process exited, code=exited, status=1/FAILURE 19:51:15 systemd[1]: Failed to start Set console font and keymap. 19:51:15 systemd[1]: console-setup.service: Unit entered failed state. 19:51:15 systemd[1]: console-setup.service: Failed with result 'exit-code'. ... /bin/setupcon has a lovely tempfile function that looks like: if \ TMPFILE=`mktemp /tmp/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /run/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /dev/.tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp /lib/init/rw/tmpkbd.XX 2>/dev/null` \ || TMPFILE=`mktemp 2>/dev/null` Here's our edited IRC conversation on the bug: <@vorlon> so I do think you're being hit by the tmp cleaner <@vorlon> also this seems like bad pathological default behavior for the tmp cleaner, to delete files that have just been created <@vorlon> but we should fix console-setup to not rely on /tmp <@vorlon> and I prefer that we do that instead of trying to fiddle with the ordering of the systemd units on startup <@vorlon> i.e. console-setup has an undeclared dependency on systemd-tmpfiles-clean; let's remove the dependency instead of declaring it <@vorlon> are you failing the race more often now than in the past? <@rcj>vorlon: it feels like it's failing more often but I don't have data to answer that. <@vorlon> are we shipping an image with a dirty rootfs? <@vorlon> dirty in the sense that e2fsck doesn't take one look at it, say "yep, nothing to do here" and exit <@vorlon> in the sense that this is what would make dev-sda1.device slow to complete AIUI <@rcj>would filesystem resize on first boot mark it dirty? Because that will happen <@vorlon> huh good question <@vorlon> it might rcj, unclean shutdown? <@rcj>xnox: first boot ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: console-setup 1.108ubuntu15.4 ProcVersionSignature: User Name 4.15.0-1029.31~16.04.1-gcp 4.15.18 Uname: Linux 4.15.0-1029-gcp x86_64 ApportVersion: 2.20.1-0ubuntu2.18 Architecture: amd64 Date: Wed Apr 10 19:24:12 2019 PackageArchitecture: all ProcEnviron: TERM=xterm-256color PATH=(custom, no user) XDG_RUNTIME_DIR= LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: console-setup UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/console-setup/+bug/1824227/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
