[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot
please reopen if this is still an issue ** Changed in: systemd (Ubuntu) Status: Confirmed => Invalid -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1788188 Title: transient systemd ordering cycle in boot with overlayroot Status in systemd package in Ubuntu: Invalid Bug description: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the initramfs mount -t tmpfs tmpfs-root /media/root-rw mkdir /media/root-rw/overlay-workdir/_ mount --move $ROOTMNT /media/root-ro mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root mkdir $ROOTMNT/media/root-ro mkdir $ROOTMNT/media/root-rw mount --move /media/root-ro "${ROOTMNT}/media/root-ro" mount --move /media/root-rw "${ROOTMNT}/media/root-rw" # then, if 'ro' on the command line, it mounts /root read-only. mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock
Re: [Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot
@xnox, Are you suggesting the /media/root-ro entry in /etc/fstab line for /media/root-ro should have x-systemd.DefaultDependencies=no ? -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1788188 Title: transient systemd ordering cycle in boot with overlayroot Status in systemd package in Ubuntu: Confirmed Bug description: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the initramfs mount -t tmpfs tmpfs-root /media/root-rw mkdir /media/root-rw/overlay-workdir/_ mount --move $ROOTMNT /media/root-ro mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root mkdir $ROOTMNT/media/root-ro mkdir $ROOTMNT/media/root-rw mount --move /media/root-ro "${ROOTMNT}/media/root-ro" mount --move /media/root-rw "${ROOTMNT}/media/root-rw" # then, if 'ro' on the command line, it mounts /root read-only. mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering
[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot
Note to self. umount.target A special target unit that unmounts all mount and automount points on system shutdown. Mounts that shall be unmounted on system shutdown shall add Conflicts dependencies to this unit for their mount unit, which is implicitly done when DefaultDependencies=yes is set (the default). Given that /media/root-ro is a pre-requisite for /, imho it should be declared with DefaultDependencies=no. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1788188 Title: transient systemd ordering cycle in boot with overlayroot Status in systemd package in Ubuntu: Confirmed Bug description: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the initramfs mount -t tmpfs tmpfs-root /media/root-rw mkdir /media/root-rw/overlay-workdir/_ mount --move $ROOTMNT /media/root-ro mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root mkdir $ROOTMNT/media/root-ro mkdir $ROOTMNT/media/root-rw mount --move /media/root-ro "${ROOTMNT}/media/root-ro" mount --move /media/root-rw "${ROOTMNT}/media/root-rw" # then, if 'ro' on the command line, it mounts /root read-only. mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start
[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot
I uploaded open-iscsi_2.0.874-5ubuntu8 which has the debug code that we were toying with included. It also includes a fix for FTBFS bug 1791154. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1788188 Title: transient systemd ordering cycle in boot with overlayroot Status in systemd package in Ubuntu: Confirmed Bug description: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the initramfs mount -t tmpfs tmpfs-root /media/root-rw mkdir /media/root-rw/overlay-workdir/_ mount --move $ROOTMNT /media/root-ro mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root mkdir $ROOTMNT/media/root-ro mkdir $ROOTMNT/media/root-rw mount --move /media/root-ro "${ROOTMNT}/media/root-ro" mount --move /media/root-rw "${ROOTMNT}/media/root-rw" # then, if 'ro' on the command line, it mounts /root read-only. mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic
[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot
I've hit that 'retrigger' button 2 more times and have not seen failures. I just did again. Don't know what to do. :-( -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1788188 Title: transient systemd ordering cycle in boot with overlayroot Status in systemd package in Ubuntu: Confirmed Bug description: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the initramfs mount -t tmpfs tmpfs-root /media/root-rw mkdir /media/root-rw/overlay-workdir/_ mount --move $ROOTMNT /media/root-ro mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root mkdir $ROOTMNT/media/root-ro mkdir $ROOTMNT/media/root-rw mount --move /media/root-ro "${ROOTMNT}/media/root-ro" mount --move /media/root-rw "${ROOTMNT}/media/root-rw" # then, if 'ro' on the command line, it mounts /root read-only. mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource
[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot
I was pushing this through autopkgtesting on LP-Infra where we see issues with that sometimes. Unfortunately when you need an issue it does not show up, I only have 4 good cases: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic-ci-train-ppa-service-3372/cosmic/amd64/o/open-iscsi/20180823_070627_52393@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic-ci-train-ppa-service-3372/cosmic/amd64/o/open-iscsi/20180822_061335_52393@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic-ci-train-ppa-service-3372/cosmic/amd64/o/open-iscsi/20180821_215839_7dcd3@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic-ci-train-ppa-service-3372/cosmic/amd64/o/open-iscsi/20180823_083835_52393@/log.gz I won't retry further, but if anyone wants: Retrigger is: https://autopkgtest.ubuntu.com/request.cgi?release=cosmic=amd64=open-iscsi=open-iscsi%2F2.0.874-5ubuntu8%7Eppa0=ci-train-ppa-service%2Fstable-phone-overlay=ci-train-ppa-service%2F3372 And results after a while will show as updated link on: https://bileto.ubuntu.com/excuses/3372/cosmic.html -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1788188 Title: transient systemd ordering cycle in boot with overlayroot Status in systemd package in Ubuntu: Confirmed Bug description: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the
[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot
here is a pastebin of boot with systemd debugging. http://paste.ubuntu.com/p/vGTRh6WB3B/ -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1788188 Title: transient systemd ordering cycle in boot with overlayroot Status in systemd package in Ubuntu: Confirmed Bug description: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the initramfs mount -t tmpfs tmpfs-root /media/root-rw mkdir /media/root-rw/overlay-workdir/_ mount --move $ROOTMNT /media/root-ro mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root mkdir $ROOTMNT/media/root-ro mkdir $ROOTMNT/media/root-rw mount --move /media/root-ro "${ROOTMNT}/media/root-ro" mount --move /media/root-rw "${ROOTMNT}/media/root-rw" # then, if 'ro' on the command line, it mounts /root read-only. mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided On
[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot
I pushed a branch up to https://code.launchpad.net/~smoser/ubuntu/+source/open-iscsi/+git/open-iscsi/+ref/bug/1788188-debug that Christian and I are trying to get some additional debug information out of a failure. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1788188 Title: transient systemd ordering cycle in boot with overlayroot Status in systemd package in Ubuntu: Confirmed Bug description: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the initramfs mount -t tmpfs tmpfs-root /media/root-rw mkdir /media/root-rw/overlay-workdir/_ mount --move $ROOTMNT /media/root-ro mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root mkdir $ROOTMNT/media/root-ro mkdir $ROOTMNT/media/root-rw mount --move /media/root-ro "${ROOTMNT}/media/root-ro" mount --move /media/root-rw "${ROOTMNT}/media/root-rw" # then, if 'ro' on the command line, it mounts /root read-only. mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start [
[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot
While it might seem unimportant "only" hitting some open-iscsi tests we can't be sure how many cases and how many people are hit by this every now and then. So I think this really is more important than it might seem at first. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1788188 Title: transient systemd ordering cycle in boot with overlayroot Status in systemd package in Ubuntu: Confirmed Bug description: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs /ext4 defaults0 0 #LABEL=UEFI /boot/efi vfatdefaults0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the initramfs mount -t tmpfs tmpfs-root /media/root-rw mkdir /media/root-rw/overlay-workdir/_ mount --move $ROOTMNT /media/root-ro mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root mkdir $ROOTMNT/media/root-ro mkdir $ROOTMNT/media/root-rw mount --move /media/root-ro "${ROOTMNT}/media/root-ro" mount --move /media/root-rw "${ROOTMNT}/media/root-rw" # then, if 'ro' on the command line, it mounts /root read-only. mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start [