I can confirm this affects all Ubuntu >= 15.04 installations, including
Ubuntu 16.04 (LTS).

> I am reasonably sure that this is not a kernel issue but a boot script
dependency issue...

That's absolutely the case. Your intuition has not led you astray.

> Hopefully someone can point me in the right direction?

To no one's surprise, this is a high-level system.d issue rather than a
low-level kernel issue. Ergo Ubuntu >= 15.04, the release at which
Ubuntu switched from Upstart to system.d.

This issue is a significant show-stopper, particularly for users on
newer UEFI-based systems. Triggering this issue is disturbingly trivial
on such systems. Explicitly editing the "/etc/fstab" file with superuser
permissions is *NOT* required to trigger this issue. Formatting a single
device with the builtin GUI-driven disk utility ("Disks") is all it
takes. And when it takes, most users will have no recourse but to format
their root filesystem and reinstall.

Here's the use case my better half stumbled into this morning. During
Ubuntu installation, users may elect to automount devices to arbitrary
mountpoints when selecting a custom partition scheme. When this is done
on UEFI-based systems, the resulting "/etc/fstab" entries resemble:

UUID=050e1e34-39e6-4072-a03e-ae0bf90ba13a /home/waluigi/wah! ext4
defaults 0 2

If any such device is subsequently reformatted (e.g., via the stock
"Disks" utility), that device's UUID will be arbitrarily changed,
invalidating the UUID previously recorded for that device in
"/etc/fstab". Everything will superficially appear to behave as
expected. On the next reboot, however, the end user will be presented
with the now-infamous Purple Screen of Death. No error messages
(...human-readable or otherwise) will appear, obscuring the underlying
issue.

Attempting to login with "safe mode" fails. Attempting to login to a
root terminal succeeds, albeit only after a few dispiriting rounds of
graphical corruption and terminal cursors disappearing. If the user
successfully navigates this shameful gauntlet of pain *AND* is
sufficiently familiar with low-level system administration to navigate
the crucible by geek fire that is the CLI, there still exists no
explicit indication of the core issue. (Command-line-fu: why have you
betrayed me?)

No relevant warnings or errors appear in either "dmesg" or "journalctl"
output *OR* in "/var/log" logfiles. The only clear indicator that I
could grep across was a single line of "systemctl -a" output showing the
status of some ignorable system service unit with a non-human-readable
name to be "loaded inactive dead", which seemed dimly suspicious.
Further hours of cursing and grepping yielded the final culprit. My
precious sanity. I have less of it now.

Non-tech-savvy users confronted with this problem will probably just
vomit, format, and reinstall. As a Gentoo-hardened Disciple of the
Command-line Faith with fifteen caffeine-addled years experience in
Silicon Valley startup monoculture, I feel overly confident in betting
that even the most battle-weary code warriors will hand in their Richard
M. Stallman fan club memberships when face-planting into this epic fail.
Consider escalating this issue's importance to at least "High" –
possibly even "Critical."

System.d's "NetworkManager.service" is the culprit. For reasons patently
unclear to me [read: "Why I Hate system.d and You Should Too"], this
service unconditionally transitively depends on the mountability of
*ALL* automounts listed in "/etc/fstab" -- regardless of whether these
automounts are actually used and hence required at startup. We can
hopefully agree that "/home/waluigi/wah!" is a non-essential automount.
Waluigi begs to disagree, of course.

System.d's "/usr/lib/systemd/system-generators/systemd-fstab-generator"
script appears to auto-generate one system service unit for each
mountpoint listed in "/etc/fstab". The failure of even a single such
unit is sufficient to bring the entire boot process to its RedHat-
stained knees. The fragility of system.d was already legendary. This
contemptible breakage only cements its well-deserved ill repute. For
shame, Poettering.

To be fair, this issue is the product of numerous cascading failures –
including:

* The failure of the stock "Disks" utility to detect and handle attempts to 
format UUID-style devices automounted by "/etc/fstab". On detecting such an 
attempt, this utility should either (in descending order of preference):
  * Globally search-and-replace the UUID field of all entries in "/etc/fstab" 
matching the offending device's prior UUID with the new UUID allocated for that 
device. In other words, "sed -i -e 's/^UUID=old-uuid/UUID=new-uuid' 
/etc/fstab". Always "sed". (Ideal.)
  * Display a non-fatal warning that formatting this device will probably 
render the entire system unbootable unless the user manually modifies 
"/etc/fstab", but permissively permit them to do it anyway. (Non-ideal, but 
this is Linux. Every user should have the right to catastrophically betray 
themselves.)
  * Display a fatal error prohibiting the user from formatting this device. 
(Don't do this.)
* The failure of the Ubuntu startup process to display coherent warnings or 
errors. I (and possibly other experienced users bludgeoned by this) would 
happily settle for a single line in either "dmesg" or "journalctl" output 
indicating the underlying issue. Ideally, however, boot failures should be 
detected by the startup process and:
  * Rendered into human-readable GUI-based error dialogues or CLI-based error 
messages. In either case, the user should be presented with some digestible 
snippet of text for subsequent feeding to Google. Boot failures shouldn't 
necessitate a degree in post-graduate Computer Science.
  * The user should be presented with the option of dropping into a CLI-based 
login prompt or (failing that) at least into a rescue terminal with superuser 
privileges. For safety, all currently mounted filesystems should be remounted 
as read-only. Because something is better than no thing. No thing and the 
smugness of the Purple Screen of Death is all we currently got.
* The failure of system.d. Oh, the complete failure of system.d. I don't even. 
On the bright side, you can only go up from rock-bottom destitution. Let's 
begin by belabouring the hopefully obvious:
  * The failure to mount optional automounts should *NOT* render the system 
unbootable. (This goes without saying. Apparently, it needs saying.)
  * The system.d journal ("journalctl") should explicitly log failing units. 
Bizarrely, it doesn't in this case. The only indication of imminent badness is 
a discontinuity after system.d attempts to start the "NetworkManager.service". 
Although no explicit error or warning is logged, the startup process silently 
fails at that service and then auto-restarts itself from the beginning... until 
slamming into that service again, at which point the cycle of depressing 
incompetence recapitulates itself. Reliably logging errors and warnings would 
seem to be the principal use case for "journalctl", but wut do I know? (See no 
error, hear no error, log no error.)
  * To beat the dead horse, all autogenerated system service units mounting 
"/etc/fstab"-listed mountpoints should be strictly optional. Exceptions might 
include the obvious system prerequisites (e.g., "/usr", "/var"). Since the root 
filesystem has (by definition) already been mounted and system.d has (by 
definition) already been exec-ed into the init process via PID 1, the system 
should at least be nominally bootable. There's little to no justification for 
automount failure to completely destroy bootability. Indeed, I'd probably 
advise all such units to be optional – regardless of whether they appear to be 
prerequisites or not. If "/usr" isn't mountable, for example, error messages to 
that effect will hopefully already be logged to "dmesg" and/or "journalctl" 
and/or printed to the current terminal. Moreover, subsequent units will 
implicitly fail by virtue of "/usr" going AWOL. System.d prematurely halting 
the startup process at the first unmountable automount adds no tangible benefit 
to anything. Let the startup process proceed with fingers and tentacles crossed 
in the event of automount failure and may the succubus take the hindmost.
  * To that effect, all other units currently hard-requiring automount units 
("local.target", possibly?) should be revised to use 
"Wants=INSERT-MOUNTPOINT-UNIT-NAME-HERE" rather than 
"Requires=INSERT-MOUNTPOINT-UNIT-NAME-HERE".

In short, I mourn the untimely death of Upstart.

It'd be awesome-sauce if either the original submitter (Mark) or another
contributor with sufficient privileges could:

* Change the affected packages to at least "system.d" (and possibly 
"gnome-disks" as well, if that indeed be the disk utility that Ubuntu currently 
leverages).
* Escalate the importance to at least "High" (and possibly "Critical").

Life in the code trenches. It doesn't get uglier than this.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1463120

Title:
  Failure to boot if fstab disk mounts fail

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1463120/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to