Re: [systemd-devel] Root remaining read-only after boot, no obvious reason why

[Dave Re]

https://freedesktop.org/wiki/Software/systemd/Debugging/

Unfortunately, setting systemd.log_level=debug, and then capturing the full 
output doesn't expose anything useful. No errors or indications about why 
systemd-remount-fs.service isn't being run, that I can see. It looks an awful 
lot like the symptoms called out here 
https://github.com/systemd/systemd/issues/5358 - but no local-fs.target wants 
are masked, in my case, so it's not the same.
[https://opengraph.githubassets.com/ab8e42c6ce897cd924fdbf909b821c5a2672314ed89248ccb9fb8250cff4ba10/systemd/systemd/issues/5358]
Prioritize systemd-remount-fs for local-fs.target · Issue #5358 · 
systemd/systemd
Submission type Bug report Request for enhancement (RFE) systemd version the 
issue has been seen with systemd 232 Used distribution Arch Linux In case of 
bug report: Expected behaviour you didn't s...
github.com


Dave Re
Manager, DevOps Engineering
www.imprivata.com
20 CityPoint, 6th Floor
​480 Totten Pond Road
​Waltham, MA  02451
The information transmitted, including attachments, is intended only for the 
person(s) or entity to which it is addressed and may contain confidential 
and/or privileged material. Any review, retransmission, dissemination or other 
use of, or taking of any action in reliance upon this information by persons or 
entities other than the intended recipient is prohibited. If you received this 
in error, please contact the sender and destroy any copies of this information. 
 [ImprivataV12018]  

Re: [systemd-devel] Root remaining read-only after boot, no obvious reason why

[Brian Reichert]

On Tue, May 16, 2023 at 05:26:00PM +, Dave Re wrote:

This is not a response to your issue, but I wanted to reach out and say that
I also work at Imprivata, in the Spirit group. :)

  breich...@imprivata.com

And I'm stuck with systemd for the now-ancient SLES12 distibution...

And I have a separate CentOS 7 distribution that I'm kepeing alive, as
well...

We could share notes, at some point...

> Dave Re
> Manager, DevOps Engineering
> www.imprivata.com
> 20 CityPoint, 6th Floor
> ???480 Totten Pond Road
> ???Waltham, MA  02451
> The information transmitted, including attachments, is intended only for the 
> person(s) or entity to which it is addressed and may contain confidential 
> and/or privileged material. Any review, retransmission, dissemination or 
> other use of, or taking of any action in reliance upon this information by 
> persons or entities other than the intended recipient is prohibited. If you 
> received this in error, please contact the sender and destroy any copies of 
> this information.  [ImprivataV12018]  

-- 
Brian Reichert  
BSD admin/developer at large

Re: [systemd-devel] Root remaining read-only after boot, no obvious reason why

[Zbigniew Jędrzejewski-Szmek]

On Tue, May 16, 2023 at 05:26:00PM +, Dave Re wrote:
> One step I haven't taken is turning on debug level logging in systemd - not 
> sure what to expect (or not expect) out of it, but I guess that's my next 
> step. I'd welcome any pointers to debugging resources, etc, but realize this 
> is an older version of systemd.

https://freedesktop.org/wiki/Software/systemd/Debugging/

Zbyszek

Re: [systemd-devel] Root remaining read-only after boot, no obvious reason why

[Dave Re]

Hmm, so we stopped enabling systemd-remount-fs.service in this way back in 2018
(https://github.com/systemd/systemd/commit/9b69569d2c). At this point,
that's ancient history for us ;)

We're unfortunately stuck to the RHEL 8 choice for systemd, yeah... ancient for 
sure. One step I haven't taken is turning on debug level logging in systemd - 
not sure what to expect (or not expect) out of it, but I guess that's my next 
step. I'd welcome any pointers to debugging resources, etc, but realize this is 
an older version of systemd.


Dave Re
Manager, DevOps Engineering
www.imprivata.com
20 CityPoint, 6th Floor
​480 Totten Pond Road
​Waltham, MA  02451
The information transmitted, including attachments, is intended only for the 
person(s) or entity to which it is addressed and may contain confidential 
and/or privileged material. Any review, retransmission, dissemination or other 
use of, or taking of any action in reliance upon this information by persons or 
entities other than the intended recipient is prohibited. If you received this 
in error, please contact the sender and destroy any copies of this information. 
 [ImprivataV12018]  

Re: [systemd-devel] Root remaining read-only after boot, no obvious reason why

[Zbigniew Jędrzejewski-Szmek]

On Tue, May 16, 2023 at 02:37:58PM +, Dave Re wrote:
> 
> Hi, Zbyszek,
> 
> The kernel default is to mount root ro, so unless there's some configuration 
> to
> mount or remount it rw, it'll stay ro. Generally this is configured in two
> places 'rw' on the kernel command line and 'rw' in /etc/fstab. Do you have
> one or both of those?
> Yes. In this case, the kernel command line in grub contains the usual "ro" 
> option, but /etc/fstab has an entry for root with "defaults" as the sole 
> option. Default options are theoretically "rw, suid, dev, exec, auto, nouser, 
> and async". To verify, I placed that option list in /etc/fstab for root, and 
> saw the same behavior on reboot, so I eliminated the fstab entry as a 
> potential issue.

Yes, systemd-remount-fs just calls 'mount -o remount /', and that remounts rw if
'ro' has not been specified. So the problem seems to be elsewhere.

> Also related - the device is identified in fstab by UUID, not by label or 
> partition. I verified the UUID of the device still matches the UUID in fstab, 
> etc.

This shouldn't matter. For the remount, we look for the file system by
mountpoint.

> If I change the kernel command line in grub to use "rw", root mounts rw as 
> expected.
> If a remount is done, it'll be done by systemd-remount-fs.service. Do you
> see it in the logs?
> Ah, I misunderstood initrd-switch-root - thanks. On a system that 
> successfully boots with root mounted rw, I don't see anything about 
> systemd-remount-fs.service until it's shutdown, when the service is stopped, 
> but systemd was ratelimited by the kernel on that boot, so a startup message 
> may have been missed. But - I can see that systemd-remount-fs.service is 
> Active/exited (via systemctl status command).
> 
> On the problem system, systemd-remount-fs.service is inactive/dead - and 
> shows the same status if I reboot with "rw" on the kernel command line (which 
> would be expected, given our symptoms, I think?).

OK, so this is the problem. It seems that the service wasn't started at all.

> On both systems, it's linked in /usr/lib/systemd/system/local-fs.target.wants 
> - so, should be run as part of getting to that target, if I understand 
> correctly? What should my next step be to determine why it's not being run?

Yes.

Hmm, so we stopped enabling systemd-remount-fs.service in this way back in 2018
(https://github.com/systemd/systemd/commit/9b69569d2c). At this point,
that's ancient history for us ;)

Zbyszek

Re: [systemd-devel] Root remaining read-only after boot, no obvious reason why

[Zbigniew Jędrzejewski-Szmek]

On Mon, May 15, 2023 at 10:21:27PM +, Dave Re wrote:
> 
> Hi, all,
> 
> Hoping to get some assistance root causing this issue. We have a set of 
> CentOS 7 based systems in AWS that we're using the leapp tool to migrate to 
> AlmaLinux 8.7. Following migration, the root filesystem is ending up being 
> mounted read-only, and I'm having a devil of a time figuring out why. There 
> are no helpful (to me) messages in dmesg, or on the console during boot. Root 
> on these machines is xfs - the filesystems appear to be intact, and running 
> "sudo mount -o rw,remount /" following boot succeeds straight away.

The kernel default is to mount root ro, so unless there's some configuration to
mount or remount it rw, it'll stay ro. Generally this is configured in two
places 'rw' on the kernel command line and 'rw' in /etc/fstab. Do you have
one or both of those?

If a remount is done, it'll be done by systemd-remount-fs.service. Do you
see it in the logs?

Zbyszek