On Tue, May 16, 2023 at 02:37:58PM +0000, Dave Re wrote: > > Hi, Zbyszek, > > The kernel default is to mount root ro, so unless there's some configuration > to > mount or remount it rw, it'll stay ro. Generally this is configured in two > places 'rw' on the kernel command line and 'rw' in /etc/fstab. Do you have > one or both of those? > Yes. In this case, the kernel command line in grub contains the usual "ro" > option, but /etc/fstab has an entry for root with "defaults" as the sole > option. Default options are theoretically "rw, suid, dev, exec, auto, nouser, > and async". To verify, I placed that option list in /etc/fstab for root, and > saw the same behavior on reboot, so I eliminated the fstab entry as a > potential issue.
Yes, systemd-remount-fs just calls 'mount -o remount /', and that remounts rw if 'ro' has not been specified. So the problem seems to be elsewhere. > Also related - the device is identified in fstab by UUID, not by label or > partition. I verified the UUID of the device still matches the UUID in fstab, > etc. This shouldn't matter. For the remount, we look for the file system by mountpoint. > If I change the kernel command line in grub to use "rw", root mounts rw as > expected. > If a remount is done, it'll be done by systemd-remount-fs.service. Do you > see it in the logs? > Ah, I misunderstood initrd-switch-root - thanks. On a system that > successfully boots with root mounted rw, I don't see anything about > systemd-remount-fs.service until it's shutdown, when the service is stopped, > but systemd was ratelimited by the kernel on that boot, so a startup message > may have been missed. But - I can see that systemd-remount-fs.service is > Active/exited (via systemctl status command). > > On the problem system, systemd-remount-fs.service is inactive/dead - and > shows the same status if I reboot with "rw" on the kernel command line (which > would be expected, given our symptoms, I think?). OK, so this is the problem. It seems that the service wasn't started at all. > On both systems, it's linked in /usr/lib/systemd/system/local-fs.target.wants > - so, should be run as part of getting to that target, if I understand > correctly? What should my next step be to determine why it's not being run? Yes. Hmm, so we stopped enabling systemd-remount-fs.service in this way back in 2018 (https://github.com/systemd/systemd/commit/9b69569d2c). At this point, that's ancient history for us ;) Zbyszek
