Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Andrew Savchenko]

On Wed, 17 Feb 2016 22:26:32 -0500 Richard Yao wrote:
> On 02/17/2016 02:01 PM, Andrew Savchenko wrote:
> > On Tue, 16 Feb 2016 15:18:46 -0500 Rich Freeman wrote:
> >> On Tue, Feb 16, 2016 at 2:31 PM, Patrick Lauer  wrote:
> >>>
> >>> The failure message comes from rc-mount.sh when the list of PIDs using a
> >>> mountpoint includes "$$" which is shell shorthand for self. How can the
> >>> current shell claim to be using /usr when it is a shell that only has
> >>> dependencies in $LIBDIR ?
> >>> As far as I can tell the code at this point calls fuser -k ${list of
> >>> pids}, and fuser outputs all PIDs that still use it. I don't see how $$
> >>> can end up in there ...
> >>
> >> What does openrc do when the script fails?  Just shut down the system 
> >> anyway?
> >>
> >> If you're going to shut down the system anyway then I'd just force the
> >> read-only mount even if it is in use.  That will cause less risk of
> >> data loss than leaving it read-write.
> >>
> >> Of course, it would be better still to kill anything that could
> >> potentially be writing to it.
> > 
> > This is not always possible. Two practical cases from my experience:
> > 
> > 1) NFS v4 shares can't be unmounted if server is unreachable (even
> > with -f). If filesystem (e.g. /home or /) contains such unmounted
> > mount points, it can't be unmounted as well, because it is still in
> > use. This happens quite often if both NFS server and client are
> > running from UPS on low power event (when AC power failed and
> > battery is almost empty).
> 
> Does `umount -l /path/to/mnt` work on those?

No, if mount point is already stalled, -l is of no use. 
 
> > 2) LUKS device is in frozen state. I use this as a security
> > precaution if LUKS fails to unmount (or it takes too long), e.g.
> > due to dead mount point.
> 
> This gives me another reason to justify being a fan of integrating
> encryption directly into a filesystem

Ext4 and f2fs do this, but with limited cypersuits available.

Actually problems with LUKS are not critical: I never lost data,
integrity there and had only a small security issue. The only
failure I can remember was libgcrypt whirlpool issue (invalid
implementation on old versions and incompatible fix on new ones).

> or using ecryptfs on top of the VFS.

No, never. Not on my setups. Ecryptfs is
1) unsecure (leaks several bytes of data)
2) unreliable, as it depends on boost and other high-level C++
stuff. I had lost an ability to decrypt data because of boost XML
versioning change.

> The others were possible integrity concerns (which definitely
> happen with a frozen state,

In theory — maybe. In real life — no. I use LUKS for over 8 years,
I often had frozen shutdowns and I never had data loss there. In
terms of data integrity LUKS + ext4 is ultimate. This joint
survived even on host with failing RAM for several years.

> although mine were about excessive layering
> adding opportunities for bugs) and performance concerns from doing
> unnecessary calculations on filesystems that span multiple disks (e.g.
> each mirror member gets encrypted independently).

Ehh... why independently? Just create mdadm with proper chunk
size, put LUKS on top of it, align both LUKS and internal fs to that
size and relevant stride and performance will be optimal. On SSDs
this is harder because it is very difficult to determine erase
block size properly, but that is another issue.

Best regards,
Andrew Savchenko


pgpY2gr1jxMe4.pgp
Description: PGP signature

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Robin H. Johnson]

On Tue, Feb 16, 2016 at 12:05:33PM -0600, William Hubbs wrote:
> I have a bug that points out a significant issue with
> /etc/init.d/mount-ro in OpenRC.
> 
> Apparently, there are issues that cause it to not work properly for file
> systems which happen to be pre-mounted from an initramfs [1].
I'll look at why it's failing to change the mount to read-only, rather
than just ignore it.

I have a use case for mount-ro not previously discussed here:
kexec

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee
E-Mail : robb...@gentoo.org
GnuPG FP   : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Richard Yao]

On 02/17/2016 02:01 PM, Andrew Savchenko wrote:
> On Tue, 16 Feb 2016 15:18:46 -0500 Rich Freeman wrote:
>> On Tue, Feb 16, 2016 at 2:31 PM, Patrick Lauer  wrote:
>>>
>>> The failure message comes from rc-mount.sh when the list of PIDs using a
>>> mountpoint includes "$$" which is shell shorthand for self. How can the
>>> current shell claim to be using /usr when it is a shell that only has
>>> dependencies in $LIBDIR ?
>>> As far as I can tell the code at this point calls fuser -k ${list of
>>> pids}, and fuser outputs all PIDs that still use it. I don't see how $$
>>> can end up in there ...
>>
>> What does openrc do when the script fails?  Just shut down the system anyway?
>>
>> If you're going to shut down the system anyway then I'd just force the
>> read-only mount even if it is in use.  That will cause less risk of
>> data loss than leaving it read-write.
>>
>> Of course, it would be better still to kill anything that could
>> potentially be writing to it.
> 
> This is not always possible. Two practical cases from my experience:
> 
> 1) NFS v4 shares can't be unmounted if server is unreachable (even
> with -f). If filesystem (e.g. /home or /) contains such unmounted
> mount points, it can't be unmounted as well, because it is still in
> use. This happens quite often if both NFS server and client are
> running from UPS on low power event (when AC power failed and
> battery is almost empty).

Does `umount -l /path/to/mnt` work on those?

> 2) LUKS device is in frozen state. I use this as a security
> precaution if LUKS fails to unmount (or it takes too long), e.g.
> due to dead mount point.

This gives me another reason to justify being a fan of integrating
encryption directly into a filesystem or using ecryptfs on top of the
VFS. The others were possible integrity concerns (which definitely
happen with a frozen state, although mine were about excessive layering
adding opportunities for bugs) and performance concerns from doing
unnecessary calculations on filesystems that span multiple disks (e.g.
each mirror member gets encrypted independently).

> As far as I understand, mount-ro may be useful only if unmount
> failed, but from my practical experience, openrc just hangs forever
> in such case until UPS is shut down by battery drain.

It is useful even if mounting everything succeeds because of /. That
said, I do not think we can sanely handle every possible configuration
because someone will always come up with something new.

> Best regards,
> Andrew Savchenko
> 




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Rich Freeman]

On Wed, Feb 17, 2016 at 2:01 PM, Andrew Savchenko  wrote:
> 1) NFS v4 shares can't be unmounted if server is unreachable (even
> with -f). If filesystem (e.g. /home or /) contains such unmounted
> mount points, it can't be unmounted as well, because it is still in
> use. This happens quite often if both NFS server and client are
> running from UPS on low power event (when AC power failed and
> battery is almost empty).

Perhaps at least the behavior in this case should be configurable
(timeouts, infinite or otherwise).

If you can't contact a remote nfs server then I believe it is possible
that unwritten changes are in buffers/etc.  Depending on circumstances
I could see either pausing until the server comes back or discarding
changes and powering off could either be the appropriate behavior.

Ultimately, anything not on the disk is always at risk, and any
filesystem needs to provide for unclean shutdown to be truly robust.
A kernel panic/etc could cause loss of all data in buffers without
warning.  However, barring that we should of course engineer openrc to
shut down in the most clean manner possible, and this should include
read-only mounts for anything which can't be unmounted.

And even systemd+dracut struggles to cleanly unmount NFS roots in the
versions I'm running at least, so that is an edge case that doesn't
get much testing.

-- 
Rich

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Andrew Savchenko]

On Tue, 16 Feb 2016 15:18:46 -0500 Rich Freeman wrote:
> On Tue, Feb 16, 2016 at 2:31 PM, Patrick Lauer  wrote:
> >
> > The failure message comes from rc-mount.sh when the list of PIDs using a
> > mountpoint includes "$$" which is shell shorthand for self. How can the
> > current shell claim to be using /usr when it is a shell that only has
> > dependencies in $LIBDIR ?
> > As far as I can tell the code at this point calls fuser -k ${list of
> > pids}, and fuser outputs all PIDs that still use it. I don't see how $$
> > can end up in there ...
> 
> What does openrc do when the script fails?  Just shut down the system anyway?
> 
> If you're going to shut down the system anyway then I'd just force the
> read-only mount even if it is in use.  That will cause less risk of
> data loss than leaving it read-write.
> 
> Of course, it would be better still to kill anything that could
> potentially be writing to it.

This is not always possible. Two practical cases from my experience:

1) NFS v4 shares can't be unmounted if server is unreachable (even
with -f). If filesystem (e.g. /home or /) contains such unmounted
mount points, it can't be unmounted as well, because it is still in
use. This happens quite often if both NFS server and client are
running from UPS on low power event (when AC power failed and
battery is almost empty).

2) LUKS device is in frozen state. I use this as a security
precaution if LUKS fails to unmount (or it takes too long), e.g.
due to dead mount point.

As far as I understand, mount-ro may be useful only if unmount
failed, but from my practical experience, openrc just hangs forever
in such case until UPS is shut down by battery drain.

Best regards,
Andrew Savchenko


pgpdPGCaiu6gC.pgp
Description: PGP signature

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Richard Yao]

> On Feb 16, 2016, at 3:18 PM, Rich Freeman  wrote:
> 
>> On Tue, Feb 16, 2016 at 2:31 PM, Patrick Lauer  wrote:
>> 
>> The failure message comes from rc-mount.sh when the list of PIDs using a
>> mountpoint includes "$$" which is shell shorthand for self. How can the
>> current shell claim to be using /usr when it is a shell that only has
>> dependencies in $LIBDIR ?
>> As far as I can tell the code at this point calls fuser -k ${list of
>> pids}, and fuser outputs all PIDs that still use it. I don't see how $$
>> can end up in there ...
> 
> What does openrc do when the script fails?  Just shut down the system anyway?
> 
> If you're going to shut down the system anyway then I'd just force the
> read-only mount even if it is in use.  That will cause less risk of
> data loss than leaving it read-write.
> 
> Of course, it would be better still to kill anything that could
> potentially be writing to it.

Agreed.
> 
> -- 
> Rich
> 

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Richard Yao]

> On Feb 16, 2016, at 1:41 PM, William Hubbs  wrote:
> 
>> On Tue, Feb 16, 2016 at 01:22:13PM -0500, Rich Freeman wrote:
>>> On Tue, Feb 16, 2016 at 1:05 PM, William Hubbs  wrote:
>>> 
>>> The reason it exists is very vague to me; I think it has something to do
>>> with claims of data loss in the past.
>> 
>> Is there some other event that will cause all filesystems to be
>> remounted read-only or unmounted before shutdown?
> 
> When localmount/netmount stop they try to unmount file systems they know
> about, but they do not try to remount anything.
> 
> 
>> You definitely will want to either unmount or remount readonly all
>> filesystems prior to rebooting.  I don't think the kernel guarantees
>> that this will happen (I'd have to look at it).  Just doing a sync
>> before poweroff doesn't seem ideal - if nothing else it will leave
>> filesystems marked as dirty and likely force fscks on the next boot
>> (or at least it should - if it doesn't that is another opportunity for
>> data loss).
>> 
>> There are different ways of accomplishing this of course, but you
>> really want to have everything read-only in the end.
> 
> unmounting is easy enough; we already do that.
> 
> What I'm trying to figure out is, what to do about re-mounting file
> systems read-only.
> 
> How does systemd do this? I didn't find an equivalent of the mount-ro
> service there.

One idea proposed by systemd that is almost never used in production is to fall 
back to an initramfs environment to undo the boot process by umounting /. It 
would not surprise me if the normal case were hard coded to remount / as ro 
because you risk filesystem corruption otherwise. Journaling filesystems are 
fairly good at surviving that, but you are still taking a risk due to partial 
writes and anyone using ext2 would be taking a much bigger gamble.
> 
> William

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Richard Yao]

> On Feb 16, 2016, at 1:05 PM, William Hubbs  wrote:
> 
> All,
> 
> I have a bug that points out a significant issue with
> /etc/init.d/mount-ro in OpenRC.
> 
> Apparently, there are issues that cause it to not work properly for file
> systems which happen to be pre-mounted from an initramfs [1].
> 
> This service only exists in the Linux world; there is no equivalent in
> OpenRC for any other operating systems we support.
> 
> The reason it exists is very vague to me; I think it has something to do
> with claims of data loss in the past.
> 
> I'm asking for more specific information, and if there is none, due to
> the bug I lincluded in this message, I am considering removing this
> service in 0.21 since I can't find an equivalent anywhere else.

If you shutdown the system while ext4 or XFS is mounted rw, then there is no 
guarantee that dirty data writeout finishes and you can get situations where 
partial writes to disks from power being cut at the end of the shutdown process 
kills the filesystem.

That said, ZFS does not need to be remounted rw because it is an atomic 
filesystem. There might be data loss, but it would only be the last 5 seconds 
of unsynced data (which is safe to use) and there would be no risk of killing 
it.

> Thanks,
> 
> William
> 
> [1] https://bugs.gentoo.org/show_bug.cgi?id=573760

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Luca Barbato]

On 16/02/16 19:05, William Hubbs wrote:
> All,
> 
> I have a bug that points out a significant issue with
> /etc/init.d/mount-ro in OpenRC.
> 
> Apparently, there are issues that cause it to not work properly for file
> systems which happen to be pre-mounted from an initramfs [1].

Who is using that file system? Ideally if "we" are the last user of the
file system it should be safe to mount ro it as well.

In general this happens when there is a "too smart to fit /" filesystem
in use.

In general that means that the same stuff used in /usr to mount it
should live in the initrd...

In general deprecating split-/usr moves the problem in in supporting fat
initrds to begin with. (I guess needing a boot filesystem that is fuse
based and needs rabbitmq or postgresql might be extra fun btw).

> This service only exists in the Linux world; there is no equivalent in
> OpenRC for any other operating systems we support.

Given it is a safety feature I do not know how the other kernels achieve
the same out of box.

> The reason it exists is very vague to me; I think it has something to do
> with claims of data loss in the past.

I think any fuse-supporting system should have it for more or less
obvious reasons (see the evil example above).

lu

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Rich Freeman]

On Tue, Feb 16, 2016 at 2:31 PM, Patrick Lauer  wrote:
>
> The failure message comes from rc-mount.sh when the list of PIDs using a
> mountpoint includes "$$" which is shell shorthand for self. How can the
> current shell claim to be using /usr when it is a shell that only has
> dependencies in $LIBDIR ?
> As far as I can tell the code at this point calls fuser -k ${list of
> pids}, and fuser outputs all PIDs that still use it. I don't see how $$
> can end up in there ...

What does openrc do when the script fails?  Just shut down the system anyway?

If you're going to shut down the system anyway then I'd just force the
read-only mount even if it is in use.  That will cause less risk of
data loss than leaving it read-write.

Of course, it would be better still to kill anything that could
potentially be writing to it.

-- 
Rich

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Daniel Campbell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 02/16/2016 10:05 AM, William Hubbs wrote:
> All,
> 
> I have a bug that points out a significant issue with 
> /etc/init.d/mount-ro in OpenRC.
> 
> Apparently, there are issues that cause it to not work properly for
> file systems which happen to be pre-mounted from an initramfs [1].
> 
> This service only exists in the Linux world; there is no equivalent
> in OpenRC for any other operating systems we support.
> 
> The reason it exists is very vague to me; I think it has something
> to do with claims of data loss in the past.
> 
> I'm asking for more specific information, and if there is none, due
> to the bug I lincluded in this message, I am considering removing
> this service in 0.21 since I can't find an equivalent anywhere
> else.
> 
> Thanks,
> 
> William
> 
> [1] https://bugs.gentoo.org/show_bug.cgi?id=573760
> 

I have a LUKS-encrypted / with LVM volumes within it, and they
(apparently?) need to be remounted read-only upon shutdown. I've not
found an easy way to take a picture of the process so I can indicate
*why*, but I've seen it scroll by.

I use an initramfs to decrypt the LUKS partition and discover the LVM
volumes. I can only assume it goes read-only before shutdown/reboot
because it needs to.

Just my two cents.

- -- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQIcBAEBCAAGBQJWw4CkAAoJEAEkDpRQOeFwrUoP+wR1Poxssa29MNQOGo0INPKW
7lKTDIDYuLFHqsKbgwBClaDQSNMEDyeZxmcpw7WXYxykftJ7IXlOYP55WDK/65kQ
QvcF+p7WhsG41kmcmKWrZpaL6Qo8wI58Bo/y3i62/lsecQiuuX8bPlTwTCLBvBxW
iDlMPPMRkHoTheTZK6BPJ4F7jkHKD6LjWKSH1cTigKOtX1Sl+ZYWw6RPgAvS3TPW
9rP6v6QBxYaeM9qiEr4OPyKwiDtm0QOfLswWSxwCmRvZriED8to1n2/CQsSee29L
C04j9nfj5Ez+O8q0hDaiuTZxVZ5AJm3obGQtKpcTDdtHxAQvKvG71AY6wuOfSDh1
kWHfUSSCGv8dB+bglb1beFGyUdZYNt3baVYhmiRqtAUGCXCSOm1Vpj8XCPOGcyPc
FOZaj6iqBfSwZGn5Ah3PwTMiwH+qgPOsYzuhb1okts99DVtNQZ28IGHvOzhYpqvZ
D/loe++/Y05HTmY9S+BhaePe0yU1BKjfO4BtMSsBHcm9cOPjni6KZRa6mGRN4fZS
X7XThXuv9WoD14FLvTMf2DbLJuNivvoyHBZ5qVuSfSachDII6LxaM4Mdvm3bbT67
Bt3jPua39eRBYm0EBNKrM7wOpE2NfLtjZ7bwoaX67EvvG1TuiYYLkXHrB9arFm09
rKNCGZvRDqwDx/PgJkFd
=wWjL
-END PGP SIGNATURE-

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Patrick Lauer]

On 02/16/2016 07:05 PM, William Hubbs wrote:
> All,
>
> I have a bug that points out a significant issue with
> /etc/init.d/mount-ro in OpenRC.
>
> Apparently, there are issues that cause it to not work properly for file
> systems which happen to be pre-mounted from an initramfs [1].
I don't understand how this fails, how does mounting from the initramfs
cause issues?

The failure message comes from rc-mount.sh when the list of PIDs using a
mountpoint includes "$$" which is shell shorthand for self. How can the
current shell claim to be using /usr when it is a shell that only has
dependencies in $LIBDIR ?
As far as I can tell the code at this point calls fuser -k ${list of
pids}, and fuser outputs all PIDs that still use it. I don't see how $$
can end up in there ...

>
> This service only exists in the Linux world; there is no equivalent in
> OpenRC for any other operating systems we support.
>
> The reason it exists is very vague to me; I think it has something to do
> with claims of data loss in the past.
Yes, if you just shut down without unmounting file systems -
(1) you may throw away data in the FS cache that hasn't ended up on disk yet
(2) the filesystem has no chance to mark itself cleanly unmounted, so
you will trigger journal replay or fsck or equivalent on boot

That's why sysvinit had a random "sleep(1)" in the halt and "sleep(2)"
in the reboot function, to give computers more of a chance to shutdown
and reboot sanely.

The changes in sysvinit-2.88-r8 and later add the "-n" option:
   -n Don't sync before reboot or halt. Note that the kernel and
storage drivers may still sync.

This was added *because* we can guarantee that filesystems are
consistent enough with mount-ro. If you wish to remove it you need to
reconsider all these little details ...
>
> I'm asking for more specific information, and if there is none, due to
> the bug I lincluded in this message, I am considering removing this
> service in 0.21 since I can't find an equivalent anywhere else.
Please don't just remove things you don't understand.
>
> Thanks,
>
> William
>
> [1] https://bugs.gentoo.org/show_bug.cgi?id=573760

Looking at the init script as of openrc-0.20.5:

~line32:
# Bug 381783
local rc_svcdir=$(echo $RC_SVCDIR | sed
's:/lib\(32\|64\)\?/:/lib(32|64)?/:g')
This looks relatively useless with everything migrated to /run and can
most likely be removed

~line35:
  local
m="/dev|/dev/.*|/proc|/proc.*|/sys|/sys/.*|/run|${rc_svcdir}" x= fs=
Since this is a regexp it can be cut down to something more simple - why
/dev and /dev/* when the second one is already excluded by the first
one. Also rc_svcdir is most likely a subdir of /run ...

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[William Hubbs]

On Tue, Feb 16, 2016 at 01:22:13PM -0500, Rich Freeman wrote:
> On Tue, Feb 16, 2016 at 1:05 PM, William Hubbs  wrote:
> >
> > The reason it exists is very vague to me; I think it has something to do
> > with claims of data loss in the past.
> >
> 
> Is there some other event that will cause all filesystems to be
> remounted read-only or unmounted before shutdown?

When localmount/netmount stop they try to unmount file systems they know
about, but they do not try to remount anything.


> You definitely will want to either unmount or remount readonly all
> filesystems prior to rebooting.  I don't think the kernel guarantees
> that this will happen (I'd have to look at it).  Just doing a sync
> before poweroff doesn't seem ideal - if nothing else it will leave
> filesystems marked as dirty and likely force fscks on the next boot
> (or at least it should - if it doesn't that is another opportunity for
> data loss).
> 
> There are different ways of accomplishing this of course, but you
> really want to have everything read-only in the end.

unmounting is easy enough; we already do that.

What I'm trying to figure out is, what to do about re-mounting file
systems read-only.

How does systemd do this? I didn't find an equivalent of the mount-ro
service there.

William


signature.asc
Description: Digital signature

Re: [gentoo-dev] rfc: Does OpenRC really need mount-ro

[Rich Freeman]

On Tue, Feb 16, 2016 at 1:05 PM, William Hubbs  wrote:
>
> The reason it exists is very vague to me; I think it has something to do
> with claims of data loss in the past.
>

Is there some other event that will cause all filesystems to be
remounted read-only or unmounted before shutdown?

You definitely will want to either unmount or remount readonly all
filesystems prior to rebooting.  I don't think the kernel guarantees
that this will happen (I'd have to look at it).  Just doing a sync
before poweroff doesn't seem ideal - if nothing else it will leave
filesystems marked as dirty and likely force fscks on the next boot
(or at least it should - if it doesn't that is another opportunity for
data loss).

There are different ways of accomplishing this of course, but you
really want to have everything read-only in the end.

-- 
Rich

[gentoo-dev] rfc: Does OpenRC really need mount-ro

[William Hubbs]

All,

I have a bug that points out a significant issue with
/etc/init.d/mount-ro in OpenRC.

Apparently, there are issues that cause it to not work properly for file
systems which happen to be pre-mounted from an initramfs [1].

This service only exists in the Linux world; there is no equivalent in
OpenRC for any other operating systems we support.

The reason it exists is very vague to me; I think it has something to do
with claims of data loss in the past.

I'm asking for more specific information, and if there is none, due to
the bug I lincluded in this message, I am considering removing this
service in 0.21 since I can't find an equivalent anywhere else.

Thanks,

William

[1] https://bugs.gentoo.org/show_bug.cgi?id=573760


signature.asc
Description: Digital signature