Re: [systemd-devel] Safe handling of root filesystem on shutdown
Hi Lennart, On Thu, Apr 12, 2012 at 8:46 AM, Daniel Drake d...@laptop.org wrote: The mmcblk0p2 message above suggests that / is being re-mounted readonly, and also on next boot the system no longer complains about / not being cleanly unmounted. Tested with 3 reboots to be sure. Reverting these commits seems like a good solution to me. If you go ahead with this, I'd also appreciate it if you could apply the fix to the F17 package next time you are touching things there. Bump :) Can these patches be reverted then? If it makes your life easier, I've attached a patch to do so. At this point I'd also like to get this sorted in F17 sooner rather than later. If you don't object, I'll patch this into the F17/F18 packages and submit an update once it is fixed in systemd git. Thanks, Daniel 0001-Don-t-skip-bind-mounts-on-shutdown.patch Description: Binary data ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Safe handling of root filesystem on shutdown
'Twas brillig, and Daniel Drake at 11/04/12 17:33 did gyre and gimble: On Wed, Apr 11, 2012 at 9:40 AM, Lennart Poettering lenn...@poettering.net wrote: So on shutdown after stopping all services we execute systemd-shutdown as PID 1 replacing the normal systemd process. This is useful to drop all references to files on disk, so that we can remount the disk r/o even on upgrades. systemd-shutdown is basically a single loop that tries to umount/read-only mount all file systems it finds as long as this changes the list of active mounts. This code also disables all swaps and detachs DM/loop devices in the same loop. Thanks as always for the fast and good explanation! With that pointer, I found the problem, see below. We do have a bit of a strange fs-layout, where our root fs is kept inside /versions/pristine/X on the root partition. The initramfs takes care of this with some bind-mount and chroot tricks so that it looks 'normal' afterwards, but maybe something along these lines is confusing systemd. chroot()? Meh, you should not use chroot for these kinds of things... Actually, we don't use chroot directly. Here's what happens: dracut mounts the root fs at /sysroot, then in a pre-pivot dracut trigger OLPC does: mkdir /vsysroot mount --bind /sysroot/versions/run/6 /vsysroot umount /sysroot NEWROOT=/vsysroot Dracut then goes ahead and performs switch_root on $NEWROOT to pivot onto the real system. (Happy to hear advice on a nicer way to do this) When the system finishes booting, /proc/self/mountinfo looks like: http://dev.laptop.org/~dsd/20120411/mountinfo.txt Now, in systemd-shutdown we reach mount_points_list_get() in umount.c, which does: /* If we encounter a bind mount, don't try to remount * the source dir too early */ skip_ro = !streq(root, /); Hence skip_ro gets set to 1 for our / mount_points_list_remount_read_only() then ignores the / mount and leaves it as RW during shutdown. I don't really understand the reasoning for the above behaviour of bind mounts. Would it be acceptable to special-case this condition if the path in question is / so that skip_ro does not get set? Or are there other options available? Keep in mind that systemd cannot umount / (and often cannot unmount /usr either) so after it has done everything it can it should re-enter the dracut initrd to let it do some additional tidy up, i.e. properly unmounting /usr and /, deactivating LVM+crypt if they were used for / or /usr (tho' I do not think either of these is done just yet - needs work at dracut side) etc. Make sure you build dracut with --systemdsystemunitdir arguement to get the dracut-shutdown.service files built and make sure they are included on the running system. HTHs Col -- Colin Guthrie gmane(at)colin.guthr.ie http://colin.guthr.ie/ Day Job: Tribalogic Limited http://www.tribalogic.net/ Open Source: Mageia Contributor http://www.mageia.org/ PulseAudio Hacker http://www.pulseaudio.org/ Trac Hacker http://trac.edgewall.org/ ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Safe handling of root filesystem on shutdown
On Wed, 11.04.12 10:33, Daniel Drake (d...@laptop.org) wrote: Now, in systemd-shutdown we reach mount_points_list_get() in umount.c, which does: /* If we encounter a bind mount, don't try to remount * the source dir too early */ skip_ro = !streq(root, /); Hence skip_ro gets set to 1 for our / mount_points_list_remount_read_only() then ignores the / mount and leaves it as RW during shutdown. I don't really understand the reasoning for the above behaviour of bind mounts. Would it be acceptable to special-case this condition if the path in question is / so that skip_ro does not get set? Or are there other options available? I think I added this logic primarily to make the shutdown loop quiet. However I must admit that that's just a guess and since my commit message is disappointingly unconclusive about this I am a bit lost... If you revert f3accc08, do things look good for you then? Do you get any log spew on shutdown? Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Safe handling of root filesystem on shutdown
On Thu, Apr 12, 2012 at 4:56 AM, Lennart Poettering lenn...@poettering.net wrote: I think I added this logic primarily to make the shutdown loop quiet. However I must admit that that's just a guess and since my commit message is disappointingly unconclusive about this I am a bit lost... If you revert f3accc08, do things look good for you then? Do you get any log spew on shutdown? I had to revert d72238fcb34abc81aca97c5fb15888708ee937d3 first. Then I reverted f3accc08, and modified systemd-shutdown to log to kmsg so that I could see the messages before power-down. [ 441.206413] systemd-shutdown[1]: Sending SIGTERM to remaining processes... [ 441.239944] systemd-shutdown[1]: Sending SIGKILL to remaining processes... [ 441.263633] systemd-shutdown[1]: Unmounting file systems. [ 441.280554] systemd-shutdown[1]: Unmounted /var/lib/random-seed. [ 441.297471] systemd-shutdown[1]: Unmounted /var/lib/dhclient. [ 441.320312] systemd-shutdown[1]: Unmounted /var/lib/dbus. [ 441.340072] systemd-shutdown[1]: Unmounted /dev/hugepages. [ 441.355911] systemd-shutdown[1]: Unmounted /sys/kernel/debug. [ 441.372049] systemd-shutdown[1]: Unmounted /dev/mqueue. [ 441.387525] systemd-shutdown[1]: Unmounted /home. [ 441.751119] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null) [ 441.831283] systemd-shutdown[1]: Disabling swaps. [ 441.846084] systemd-shutdown[1]: Detaching loop devices. [ 441.864999] systemd-shutdown[1]: Detaching DM devices. [ 442.965933] ACPI: Preparing to enter system sleep state S5 [ 443.080153] Power down. The mmcblk0p2 message above suggests that / is being re-mounted readonly, and also on next boot the system no longer complains about / not being cleanly unmounted. Tested with 3 reboots to be sure. Reverting these commits seems like a good solution to me. If you go ahead with this, I'd also appreciate it if you could apply the fix to the F17 package next time you are touching things there. Thanks! Daniel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Safe handling of root filesystem on shutdown
On Wed, 11.04.12 09:08, Daniel Drake (d...@laptop.org) wrote: Heya, On OLPC laptops we are seeing that ext4 complains on every boot that the filesystem wasn't cleanly unmounted. Looking at systemd debug logs of a shutdown would seem to agree, I can't see where it attempts to remount / read-only as was done with sysvinit. http://dev.laptop.org/~dsd/20120411/shutdown.txt Can anyone point out how this is supposed to work - where is the code that looks after the / mount during shutdown/reboot? So on shutdown after stopping all services we execute systemd-shutdown as PID 1 replacing the normal systemd process. This is useful to drop all references to files on disk, so that we can remount the disk r/o even on upgrades. systemd-shutdown is basically a single loop that tries to umount/read-only mount all file systems it finds as long as this changes the list of active mounts. This code also disables all swaps and detachs DM/loop devices in the same loop. We do have a bit of a strange fs-layout, where our root fs is kept inside /versions/pristine/X on the root partition. The initramfs takes care of this with some bind-mount and chroot tricks so that it looks 'normal' afterwards, but maybe something along these lines is confusing systemd. chroot()? Meh, you should not use chroot for these kinds of things... Lennart -- Lennart Poettering - Red Hat, Inc. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Safe handling of root filesystem on shutdown
On Wed, Apr 11, 2012 at 9:40 AM, Lennart Poettering lenn...@poettering.net wrote: So on shutdown after stopping all services we execute systemd-shutdown as PID 1 replacing the normal systemd process. This is useful to drop all references to files on disk, so that we can remount the disk r/o even on upgrades. systemd-shutdown is basically a single loop that tries to umount/read-only mount all file systems it finds as long as this changes the list of active mounts. This code also disables all swaps and detachs DM/loop devices in the same loop. Thanks as always for the fast and good explanation! With that pointer, I found the problem, see below. We do have a bit of a strange fs-layout, where our root fs is kept inside /versions/pristine/X on the root partition. The initramfs takes care of this with some bind-mount and chroot tricks so that it looks 'normal' afterwards, but maybe something along these lines is confusing systemd. chroot()? Meh, you should not use chroot for these kinds of things... Actually, we don't use chroot directly. Here's what happens: dracut mounts the root fs at /sysroot, then in a pre-pivot dracut trigger OLPC does: mkdir /vsysroot mount --bind /sysroot/versions/run/6 /vsysroot umount /sysroot NEWROOT=/vsysroot Dracut then goes ahead and performs switch_root on $NEWROOT to pivot onto the real system. (Happy to hear advice on a nicer way to do this) When the system finishes booting, /proc/self/mountinfo looks like: http://dev.laptop.org/~dsd/20120411/mountinfo.txt Now, in systemd-shutdown we reach mount_points_list_get() in umount.c, which does: /* If we encounter a bind mount, don't try to remount * the source dir too early */ skip_ro = !streq(root, /); Hence skip_ro gets set to 1 for our / mount_points_list_remount_read_only() then ignores the / mount and leaves it as RW during shutdown. I don't really understand the reasoning for the above behaviour of bind mounts. Would it be acceptable to special-case this condition if the path in question is / so that skip_ro does not get set? Or are there other options available? Thanks, Daniel ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
