Public bug reported: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server.
The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs / ext4 defaults 0 0 #LABEL=UEFI /boot/efi vfat defaults 0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs / ext4 defaults 0 0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the initramfs mount -t tmpfs tmpfs-root /media/root-rw mkdir /media/root-rw/overlay-workdir/_ mount --move $ROOTMNT /media/root-ro mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root mkdir $ROOTMNT/media/root-ro mkdir $ROOTMNT/media/root-rw mount --move /media/root-ro "${ROOTMNT}/media/root-ro" mount --move /media/root-rw "${ROOTMNT}/media/root-rw" # then, if 'ro' on the command line, it mounts /root read-only. mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided On successful boot, we can login and see: $ find /run/systemd/ -name "*.mount" | xargs ls -l -rw-r--r-- 1 root root 322 Aug 21 13:55 /run/systemd/generator/-.mount lrwxrwxrwx 1 root root 10 Aug 21 13:55 /run/systemd/generator/local-fs.target.requires/-.mount -> ../-.mount lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-hugepages.mount -> b847fc2d06b54b1ab86a6f526d149522 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-mqueue.mount -> f991d3bfe35d4e2bb9aa86677ff31d70 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-fs-fuse-connections.mount -> a561df591a28437f990f94269fa08be9 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-config.mount -> 141e9bbaca9041e48458749e29bc58fb lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-debug.mount -> a406163f265c4b02b37e6f92ddbfc029 $ for f in $(find /run/systemd/ -name '*.mount' -type f); do echo == $f ==; cat $f; done == /run/systemd/generator/-.mount == # Automatically generated by systemd-fstab-generator [Unit] SourcePath=/etc/fstab Documentation=man:fstab(5) man:systemd-fstab-generator(8) Before=local-fs.target [Mount] Where=/ What=/media/root-ro/ Type=overlay Options=lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ -- [1] https://git.launchpad.net/cloud-initramfs-tools/tree/overlayroot/scripts/init-bottom/overlayroot Related bugs: * bug 1723183: [overlayroot] transient systemd ordering issue when using overlayroot ProblemType: Bug DistroRelease: Ubuntu 18.10 Package: systemd 237-3ubuntu10 ProcVersionSignature: Ubuntu 4.17.0-6.7-generic 4.17.9 Uname: Linux 4.17.0-6-generic x86_64 ApportVersion: 2.20.10-0ubuntu7 Architecture: amd64 Date: Tue Aug 21 14:06:24 2018 Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: QEMU Standard PC (i440FX + PIIX, 1996) ProcEnviron: TERM=vt220 PATH=(custom, no user) LANG=C.UTF-8 SHELL=/bin/bash ProcKernelCmdLine: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash SourcePackage: systemd UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 04/01/2014 dmi.bios.vendor: SeaBIOS dmi.bios.version: 1.11.1-1 dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-i440fx-cosmic dmi.modalias: dmi:bvnSeaBIOS:bvr1.11.1-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-cosmic:cvnQEMU:ct1:cvrpc-i440fx-cosmic: dmi.product.name: Standard PC (i440FX + PIIX, 1996) dmi.product.version: pc-i440fx-cosmic dmi.sys.vendor: QEMU ** Affects: systemd (Ubuntu) Importance: High Status: Confirmed ** Tags: amd64 apport-bug cosmic uec-images ** Description changed: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: - LABEL=cloudimg-rootfs / ext4 defaults 0 0 - #LABEL=UEFI /boot/efi vfat defaults 0 0 + LABEL=cloudimg-rootfs / ext4 defaults 0 0 + #LABEL=UEFI /boot/efi vfat defaults 0 0 when init takes over from the initramfs, we have - /proc/cmdline: - nomodeset iscsi_initiator=maas-enlist - iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 - iscsi_target_port=3260 iscsi_initiator=maas-enlist - ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 - root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 - overlayroot=tmpfs console=ttyS0 - ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash + /proc/cmdline: + nomodeset iscsi_initiator=maas-enlist + iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 + iscsi_target_port=3260 iscsi_initiator=maas-enlist + ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 + root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 + overlayroot=tmpfs console=ttyS0 + ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash - /etc/fstab: - # cat /etc/fstab - # - # This fstab is in an overlay. The real one can be found at - # /media/root-ro/etc/fstab - # The original entry for '/' and other mounts have been updated to be placed - # under /media/root-ro. - # To permanently modify this (or any other file), you should change-root into - # a writable view of the underlying filesystem using: - # sudo overlayroot-chroot - # - #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 - /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 - #LABEL=UEFI /boot/efi vfat defaults 0 0 + /etc/fstab: + # cat /etc/fstab + # + # This fstab is in an overlay. The real one can be found at + # /media/root-ro/etc/fstab + # The original entry for '/' and other mounts have been updated to be placed + # under /media/root-ro. + # To permanently modify this (or any other file), you should change-root into + # a writable view of the underlying filesystem using: + # sudo overlayroot-chroot + # + #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 + /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 + #LABEL=UEFI /boot/efi vfat defaults 0 0 - /root/root-ro/etc/fstab: - LABEL=cloudimg-rootfs / ext4 defaults 0 0 - #LABEL=UEFI /boot/efi vfat defaults 0 0 + /root/root-ro/etc/fstab: + LABEL=cloudimg-rootfs / ext4 defaults 0 0 + #LABEL=UEFI /boot/efi vfat defaults 0 0 - /proc/mounts: - sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 - proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 - udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 - devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 - tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 - /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 - tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 - overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 + /proc/mounts: + sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 + proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 + udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 + devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 + tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 + /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 + tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 + overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 - /proc/1/mountinfo: - 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw - 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw - 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 - 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 - 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 - 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro - 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw - 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ + /proc/1/mountinfo: + 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw + 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw + 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 + 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 + 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 + 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro + 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw + 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ + overlayroot's scripts/init-bottom/overlayroot script [1] inherits a + read-only mount of block device on mount point '$ROOTMNT' + (ROOTMNT=/root). In order to set up the overlayroot, it does the + following: - overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: + mkdir /media/root-ro /media/root-rw # in the initramfs + mount -t tmpfs tmpfs-root /media/root-rw + mkdir /media/root-rw/overlay-workdir/_ + mount --move $ROOTMNT /media/root-ro + mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root + mkdir $ROOTMNT/media/root-ro + mkdir $ROOTMNT/media/root-rw + mount --move /media/root-ro "${ROOTMNT}/media/root-ro" + mount --move /media/root-rw "${ROOTMNT}/media/root-rw" - mkdir /media/root-ro /media/root-rw # in the initramfs - mount -t tmpfs tmpfs-root /media/root-rw - mkdir /media/root-rw/overlay-workdir/_ - mount --move $ROOTMNT /media/root-ro - mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root - mkdir $ROOTMNT/media/root-ro - mkdir $ROOTMNT/media/root-rw - mount --move /media/root-ro "${ROOTMNT}/media/root-ro" - mount --move /media/root-rw "${ROOTMNT}/media/root-rw" - - # then, if 'ro' on the command line, it mounts /root read-only. - mount -o remount,ro $ROOTMNT + # then, if 'ro' on the command line, it mounts /root read-only. + mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: - [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start - [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start - [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start - [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided - + [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start + [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start + [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start + [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided On successful boot, we can login and see: $ find /run/systemd/ -name "*.mount" | xargs ls -l -rw-r--r-- 1 root root 322 Aug 21 13:55 /run/systemd/generator/-.mount lrwxrwxrwx 1 root root 10 Aug 21 13:55 /run/systemd/generator/local-fs.target.requires/-.mount -> ../-.mount lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-hugepages.mount -> b847fc2d06b54b1ab86a6f526d149522 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-mqueue.mount -> f991d3bfe35d4e2bb9aa86677ff31d70 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-fs-fuse-connections.mount -> a561df591a28437f990f94269fa08be9 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-config.mount -> 141e9bbaca9041e48458749e29bc58fb lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-debug.mount -> a406163f265c4b02b37e6f92ddbfc029 - $ for f in $(find /run/systemd/ -name '*.mount' -type f); do echo == $f ==; cat $f; done == /run/systemd/generator/-.mount == # Automatically generated by systemd-fstab-generator [Unit] SourcePath=/etc/fstab Documentation=man:fstab(5) man:systemd-fstab-generator(8) Before=local-fs.target [Mount] Where=/ What=/media/root-ro/ Type=overlay Options=lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ -- [1] https://git.launchpad.net/cloud-initramfs-tools/tree/overlayroot/scripts/init-bottom/overlayroot - Related bugs: - * bug 1723183: transient systemd ordering issue when using overlayroot + * bug 1723183: [overlayroot] transient systemd ordering issue when using overlayroot ProblemType: Bug DistroRelease: Ubuntu 18.10 Package: systemd 237-3ubuntu10 ProcVersionSignature: Ubuntu 4.17.0-6.7-generic 4.17.9 Uname: Linux 4.17.0-6-generic x86_64 ApportVersion: 2.20.10-0ubuntu7 Architecture: amd64 Date: Tue Aug 21 14:06:24 2018 Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: QEMU Standard PC (i440FX + PIIX, 1996) ProcEnviron: - TERM=vt220 - PATH=(custom, no user) - LANG=C.UTF-8 - SHELL=/bin/bash + TERM=vt220 + PATH=(custom, no user) + LANG=C.UTF-8 + SHELL=/bin/bash ProcKernelCmdLine: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash SourcePackage: systemd UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 04/01/2014 dmi.bios.vendor: SeaBIOS dmi.bios.version: 1.11.1-1 dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-i440fx-cosmic dmi.modalias: dmi:bvnSeaBIOS:bvr1.11.1-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-cosmic:cvnQEMU:ct1:cvrpc-i440fx-cosmic: dmi.product.name: Standard PC (i440FX + PIIX, 1996) dmi.product.version: pc-i440fx-cosmic dmi.sys.vendor: QEMU ** Changed in: systemd (Ubuntu) Status: New => Confirmed ** Changed in: systemd (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1788188 Title: transient systemd ordering cycle in boot with overlayroot Status in systemd package in Ubuntu: Confirmed Bug description: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs / ext4 defaults 0 0 #LABEL=UEFI /boot/efi vfat defaults 0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs / ext4 defaults 0 0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the initramfs mount -t tmpfs tmpfs-root /media/root-rw mkdir /media/root-rw/overlay-workdir/_ mount --move $ROOTMNT /media/root-ro mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root mkdir $ROOTMNT/media/root-ro mkdir $ROOTMNT/media/root-rw mount --move /media/root-ro "${ROOTMNT}/media/root-ro" mount --move /media/root-rw "${ROOTMNT}/media/root-rw" # then, if 'ro' on the command line, it mounts /root read-only. mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided On successful boot, we can login and see: $ find /run/systemd/ -name "*.mount" | xargs ls -l -rw-r--r-- 1 root root 322 Aug 21 13:55 /run/systemd/generator/-.mount lrwxrwxrwx 1 root root 10 Aug 21 13:55 /run/systemd/generator/local-fs.target.requires/-.mount -> ../-.mount lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-hugepages.mount -> b847fc2d06b54b1ab86a6f526d149522 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-mqueue.mount -> f991d3bfe35d4e2bb9aa86677ff31d70 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-fs-fuse-connections.mount -> a561df591a28437f990f94269fa08be9 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-config.mount -> 141e9bbaca9041e48458749e29bc58fb lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-debug.mount -> a406163f265c4b02b37e6f92ddbfc029 $ for f in $(find /run/systemd/ -name '*.mount' -type f); do echo == $f ==; cat $f; done == /run/systemd/generator/-.mount == # Automatically generated by systemd-fstab-generator [Unit] SourcePath=/etc/fstab Documentation=man:fstab(5) man:systemd-fstab-generator(8) Before=local-fs.target [Mount] Where=/ What=/media/root-ro/ Type=overlay Options=lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ -- [1] https://git.launchpad.net/cloud-initramfs-tools/tree/overlayroot/scripts/init-bottom/overlayroot Related bugs: * bug 1723183: [overlayroot] transient systemd ordering issue when using overlayroot ProblemType: Bug DistroRelease: Ubuntu 18.10 Package: systemd 237-3ubuntu10 ProcVersionSignature: Ubuntu 4.17.0-6.7-generic 4.17.9 Uname: Linux 4.17.0-6-generic x86_64 ApportVersion: 2.20.10-0ubuntu7 Architecture: amd64 Date: Tue Aug 21 14:06:24 2018 Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: QEMU Standard PC (i440FX + PIIX, 1996) ProcEnviron: TERM=vt220 PATH=(custom, no user) LANG=C.UTF-8 SHELL=/bin/bash ProcKernelCmdLine: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash SourcePackage: systemd UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 04/01/2014 dmi.bios.vendor: SeaBIOS dmi.bios.version: 1.11.1-1 dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-i440fx-cosmic dmi.modalias: dmi:bvnSeaBIOS:bvr1.11.1-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-cosmic:cvnQEMU:ct1:cvrpc-i440fx-cosmic: dmi.product.name: Standard PC (i440FX + PIIX, 1996) dmi.product.version: pc-i440fx-cosmic dmi.sys.vendor: QEMU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1788188/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp