I uploaded open-iscsi_2.0.874-5ubuntu8 which has the debug code that we were toying with included. It also includes a fix for FTBFS bug 1791154.
-- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1788188 Title: transient systemd ordering cycle in boot with overlayroot Status in systemd package in Ubuntu: Confirmed Bug description: open-iscsi test utilizes overlayroot to boot a cloud-image with root filesystem on a read-only iscsi server. The /etc/fstab file in the image looks like this: LABEL=cloudimg-rootfs / ext4 defaults 0 0 #LABEL=UEFI /boot/efi vfat defaults 0 0 when init takes over from the initramfs, we have /proc/cmdline: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash /etc/fstab: # cat /etc/fstab # # This fstab is in an overlay. The real one can be found at # /media/root-ro/etc/fstab # The original entry for '/' and other mounts have been updated to be placed # under /media/root-ro. # To permanently modify this (or any other file), you should change-root into # a writable view of the underlying filesystem using: # sudo overlayroot-chroot # #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0 /media/root-ro/ / overlay lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /root/root-ro/etc/fstab: LABEL=cloudimg-rootfs / ext4 defaults 0 0 #LABEL=UEFI /boot/efi vfat defaults 0 0 /proc/mounts: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 /media/root-ro ext4 ro,relatime 0 0 tmpfs-root /media/root-rw tmpfs rw,relatime 0 0 overlayroot / overlay ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ 0 0 /proc/1/mountinfo: 21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw 22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw 23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev rw,size=233748k,nr_inodes=58437,mode=755 24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=000 25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=49288k,mode=755 26 28 8:1 / /media/root-ro ro,relatime - ext4 /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro 27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw 28 0 0:24 / / ro,relatime - overlay overlayroot ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot's scripts/init-bottom/overlayroot script [1] inherits a read-only mount of block device on mount point '$ROOTMNT' (ROOTMNT=/root). In order to set up the overlayroot, it does the following: mkdir /media/root-ro /media/root-rw # in the initramfs mount -t tmpfs tmpfs-root /media/root-rw mkdir /media/root-rw/overlay-workdir/_ mount --move $ROOTMNT /media/root-ro mount -t overlay -o lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_ overlayroot /root mkdir $ROOTMNT/media/root-ro mkdir $ROOTMNT/media/root-rw mount --move /media/root-ro "${ROOTMNT}/media/root-ro" mount --move /media/root-rw "${ROOTMNT}/media/root-rw" # then, if 'ro' on the command line, it mounts /root read-only. mount -o remount,ro $ROOTMNT The script then exits, as ROOTMNT is now set up with a read-only mount of the overlayroot. All the mounts it has done have been moved under ROOTMNT. On failure systemd reports: [ 104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on -.mount/start [ 104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on media-root\x2dro.mount/start [ 104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle starting with media-root\x2dro.mount/start [ 104.137591] systemd[1]: Requested transaction contains an unfixable cyclic ordering dependency: Resource deadlock avoided On successful boot, we can login and see: $ find /run/systemd/ -name "*.mount" | xargs ls -l -rw-r--r-- 1 root root 322 Aug 21 13:55 /run/systemd/generator/-.mount lrwxrwxrwx 1 root root 10 Aug 21 13:55 /run/systemd/generator/local-fs.target.requires/-.mount -> ../-.mount lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-hugepages.mount -> b847fc2d06b54b1ab86a6f526d149522 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:dev-mqueue.mount -> f991d3bfe35d4e2bb9aa86677ff31d70 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-fs-fuse-connections.mount -> a561df591a28437f990f94269fa08be9 lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-config.mount -> 141e9bbaca9041e48458749e29bc58fb lrwxrwxrwx 1 root root 32 Aug 21 13:55 /run/systemd/units/invocation:sys-kernel-debug.mount -> a406163f265c4b02b37e6f92ddbfc029 $ for f in $(find /run/systemd/ -name '*.mount' -type f); do echo == $f ==; cat $f; done == /run/systemd/generator/-.mount == # Automatically generated by systemd-fstab-generator [Unit] SourcePath=/etc/fstab Documentation=man:fstab(5) man:systemd-fstab-generator(8) Before=local-fs.target [Mount] Where=/ What=/media/root-ro/ Type=overlay Options=lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_ -- [1] https://git.launchpad.net/cloud-initramfs-tools/tree/overlayroot/scripts/init-bottom/overlayroot Related bugs: * bug 1723183: [overlayroot] transient systemd ordering issue when using overlayroot ProblemType: Bug DistroRelease: Ubuntu 18.10 Package: systemd 237-3ubuntu10 ProcVersionSignature: Ubuntu 4.17.0-6.7-generic 4.17.9 Uname: Linux 4.17.0-6-generic x86_64 ApportVersion: 2.20.10-0ubuntu7 Architecture: amd64 Date: Tue Aug 21 14:06:24 2018 Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: QEMU Standard PC (i440FX + PIIX, 1996) ProcEnviron: TERM=vt220 PATH=(custom, no user) LANG=C.UTF-8 SHELL=/bin/bash ProcKernelCmdLine: nomodeset iscsi_initiator=maas-enlist iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 overlayroot=tmpfs console=ttyS0 ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash SourcePackage: systemd UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 04/01/2014 dmi.bios.vendor: SeaBIOS dmi.bios.version: 1.11.1-1 dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-i440fx-cosmic dmi.modalias: dmi:bvnSeaBIOS:bvr1.11.1-1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-cosmic:cvnQEMU:ct1:cvrpc-i440fx-cosmic: dmi.product.name: Standard PC (i440FX + PIIX, 1996) dmi.product.version: pc-i440fx-cosmic dmi.sys.vendor: QEMU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1788188/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
