[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot

2021-06-30 Thread Dan Streetman
please reopen if this is still an issue

** Changed in: systemd (Ubuntu)
   Status: Confirmed => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1788188

Title:
  transient systemd ordering cycle in boot with overlayroot

Status in systemd package in Ubuntu:
  Invalid

Bug description:
  open-iscsi test utilizes overlayroot to boot a cloud-image with root
  filesystem on a read-only iscsi server.

  The /etc/fstab file in the image looks like this:
    LABEL=cloudimg-rootfs   /ext4   defaults0 0
    #LABEL=UEFI /boot/efi   vfatdefaults0 0

  when init takes over from the initramfs, we have

    /proc/cmdline:
  nomodeset iscsi_initiator=maas-enlist
  iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
  iscsi_target_port=3260 iscsi_initiator=maas-enlist
  ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
  
root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
  overlayroot=tmpfs console=ttyS0
  ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

    /etc/fstab:
  # cat /etc/fstab
  #
  #  This fstab is in an overlay. The real one can be found at
  #  /media/root-ro/etc/fstab
  #  The original entry for '/' and other mounts have been updated to be 
placed
  #  under /media/root-ro.
  #  To permanently modify this (or any other file), you should change-root 
into
  #  a writable view of the underlying filesystem using:
  #  sudo overlayroot-chroot
  #
  #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
  #LABEL=UEFI /boot/efi vfat defaults 0 0

   /root/root-ro/etc/fstab:
  LABEL=cloudimg-rootfs   /ext4   defaults0 0
  #LABEL=UEFI /boot/efi   vfatdefaults0 0

   /proc/mounts:
     sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
     proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
     udev /dev devtmpfs 
rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
     devpts /dev/pts devpts 
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
     tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
     /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 
/media/root-ro ext4 ro,relatime 0 0
     tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
     overlayroot / overlay 
ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 0 0

   /proc/1/mountinfo:
     21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
     22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
     23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev 
rw,size=233748k,nr_inodes=58437,mode=755
     24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts 
rw,gid=5,mode=620,ptmxmode=000
     25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs 
rw,size=49288k,mode=755
     26 28 8:1 / /media/root-ro ro,relatime - ext4 
/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
     27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
     28 0 0:24 / / ro,relatime - overlay overlayroot 
ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

  overlayroot's scripts/init-bottom/overlayroot script [1] inherits a
  read-only mount of block device on mount point '$ROOTMNT'
  (ROOTMNT=/root). In order to set up the overlayroot, it does the
  following:

   mkdir /media/root-ro /media/root-rw # in the initramfs
   mount -t tmpfs tmpfs-root /media/root-rw
   mkdir /media/root-rw/overlay-workdir/_
   mount --move $ROOTMNT /media/root-ro
   mount -t overlay -o 
lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 overlayroot /root
   mkdir $ROOTMNT/media/root-ro
   mkdir $ROOTMNT/media/root-rw
   mount --move /media/root-ro "${ROOTMNT}/media/root-ro"
   mount --move /media/root-rw "${ROOTMNT}/media/root-rw"

   # then, if 'ro' on the command line, it mounts /root read-only.
   mount -o remount,ro $ROOTMNT

  The script then exits, as ROOTMNT is now set up with a read-only mount
  of the overlayroot.  All the mounts it has done have been moved
  under ROOTMNT.

  On failure systemd reports:
     [  104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
-.mount/start
     [  104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on 
media-root\x2dro.mount/start
     [  104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle 
starting with media-root\x2dro.mount/start
     [  104.137591] systemd[1]: Requested transaction contains an unfixable 
cyclic ordering dependency: Resource deadlock 

Re: [Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot

2018-09-13 Thread Scott Moser
@xnox,
Are you suggesting the /media/root-ro entry in /etc/fstab line for
/media/root-ro should have x-systemd.DefaultDependencies=no ?

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1788188

Title:
  transient systemd ordering cycle in boot with overlayroot

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  open-iscsi test utilizes overlayroot to boot a cloud-image with root
  filesystem on a read-only iscsi server.

  The /etc/fstab file in the image looks like this:
    LABEL=cloudimg-rootfs   /ext4   defaults0 0
    #LABEL=UEFI /boot/efi   vfatdefaults0 0

  when init takes over from the initramfs, we have

    /proc/cmdline:
  nomodeset iscsi_initiator=maas-enlist
  iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
  iscsi_target_port=3260 iscsi_initiator=maas-enlist
  ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
  
root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
  overlayroot=tmpfs console=ttyS0
  ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

    /etc/fstab:
  # cat /etc/fstab
  #
  #  This fstab is in an overlay. The real one can be found at
  #  /media/root-ro/etc/fstab
  #  The original entry for '/' and other mounts have been updated to be 
placed
  #  under /media/root-ro.
  #  To permanently modify this (or any other file), you should change-root 
into
  #  a writable view of the underlying filesystem using:
  #  sudo overlayroot-chroot
  #
  #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
  #LABEL=UEFI /boot/efi vfat defaults 0 0

   /root/root-ro/etc/fstab:
  LABEL=cloudimg-rootfs   /ext4   defaults0 0
  #LABEL=UEFI /boot/efi   vfatdefaults0 0

   /proc/mounts:
     sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
     proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
     udev /dev devtmpfs 
rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
     devpts /dev/pts devpts 
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
     tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
     /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 
/media/root-ro ext4 ro,relatime 0 0
     tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
     overlayroot / overlay 
ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 0 0

   /proc/1/mountinfo:
     21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
     22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
     23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev 
rw,size=233748k,nr_inodes=58437,mode=755
     24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts 
rw,gid=5,mode=620,ptmxmode=000
     25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs 
rw,size=49288k,mode=755
     26 28 8:1 / /media/root-ro ro,relatime - ext4 
/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
     27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
     28 0 0:24 / / ro,relatime - overlay overlayroot 
ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

  overlayroot's scripts/init-bottom/overlayroot script [1] inherits a
  read-only mount of block device on mount point '$ROOTMNT'
  (ROOTMNT=/root). In order to set up the overlayroot, it does the
  following:

   mkdir /media/root-ro /media/root-rw # in the initramfs
   mount -t tmpfs tmpfs-root /media/root-rw
   mkdir /media/root-rw/overlay-workdir/_
   mount --move $ROOTMNT /media/root-ro
   mount -t overlay -o 
lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 overlayroot /root
   mkdir $ROOTMNT/media/root-ro
   mkdir $ROOTMNT/media/root-rw
   mount --move /media/root-ro "${ROOTMNT}/media/root-ro"
   mount --move /media/root-rw "${ROOTMNT}/media/root-rw"

   # then, if 'ro' on the command line, it mounts /root read-only.
   mount -o remount,ro $ROOTMNT

  The script then exits, as ROOTMNT is now set up with a read-only mount
  of the overlayroot.  All the mounts it has done have been moved
  under ROOTMNT.

  On failure systemd reports:
     [  104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
-.mount/start
     [  104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on 
media-root\x2dro.mount/start
     [  104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle 
starting with media-root\x2dro.mount/start
     [  104.137591] systemd[1]: Requested transaction contains an unfixable 
cyclic ordering 

[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot

2018-09-13 Thread Dimitri John Ledkov
Note to self.

   umount.target
   A special target unit that unmounts all mount and automount points 
on system shutdown.

   Mounts that shall be unmounted on system shutdown shall add 
Conflicts dependencies to this unit for their mount unit, which is implicitly 
done
   when DefaultDependencies=yes is set (the default).

Given that /media/root-ro is a pre-requisite for /, imho it should be
declared with DefaultDependencies=no.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1788188

Title:
  transient systemd ordering cycle in boot with overlayroot

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  open-iscsi test utilizes overlayroot to boot a cloud-image with root
  filesystem on a read-only iscsi server.

  The /etc/fstab file in the image looks like this:
    LABEL=cloudimg-rootfs   /ext4   defaults0 0
    #LABEL=UEFI /boot/efi   vfatdefaults0 0

  when init takes over from the initramfs, we have

    /proc/cmdline:
  nomodeset iscsi_initiator=maas-enlist
  iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
  iscsi_target_port=3260 iscsi_initiator=maas-enlist
  ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
  
root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
  overlayroot=tmpfs console=ttyS0
  ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

    /etc/fstab:
  # cat /etc/fstab
  #
  #  This fstab is in an overlay. The real one can be found at
  #  /media/root-ro/etc/fstab
  #  The original entry for '/' and other mounts have been updated to be 
placed
  #  under /media/root-ro.
  #  To permanently modify this (or any other file), you should change-root 
into
  #  a writable view of the underlying filesystem using:
  #  sudo overlayroot-chroot
  #
  #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
  #LABEL=UEFI /boot/efi vfat defaults 0 0

   /root/root-ro/etc/fstab:
  LABEL=cloudimg-rootfs   /ext4   defaults0 0
  #LABEL=UEFI /boot/efi   vfatdefaults0 0

   /proc/mounts:
     sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
     proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
     udev /dev devtmpfs 
rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
     devpts /dev/pts devpts 
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
     tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
     /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 
/media/root-ro ext4 ro,relatime 0 0
     tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
     overlayroot / overlay 
ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 0 0

   /proc/1/mountinfo:
     21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
     22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
     23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev 
rw,size=233748k,nr_inodes=58437,mode=755
     24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts 
rw,gid=5,mode=620,ptmxmode=000
     25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs 
rw,size=49288k,mode=755
     26 28 8:1 / /media/root-ro ro,relatime - ext4 
/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
     27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
     28 0 0:24 / / ro,relatime - overlay overlayroot 
ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

  overlayroot's scripts/init-bottom/overlayroot script [1] inherits a
  read-only mount of block device on mount point '$ROOTMNT'
  (ROOTMNT=/root). In order to set up the overlayroot, it does the
  following:

   mkdir /media/root-ro /media/root-rw # in the initramfs
   mount -t tmpfs tmpfs-root /media/root-rw
   mkdir /media/root-rw/overlay-workdir/_
   mount --move $ROOTMNT /media/root-ro
   mount -t overlay -o 
lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 overlayroot /root
   mkdir $ROOTMNT/media/root-ro
   mkdir $ROOTMNT/media/root-rw
   mount --move /media/root-ro "${ROOTMNT}/media/root-ro"
   mount --move /media/root-rw "${ROOTMNT}/media/root-rw"

   # then, if 'ro' on the command line, it mounts /root read-only.
   mount -o remount,ro $ROOTMNT

  The script then exits, as ROOTMNT is now set up with a read-only mount
  of the overlayroot.  All the mounts it has done have been moved
  under ROOTMNT.

  On failure systemd reports:
     [  104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
-.mount/start

[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot

2018-09-06 Thread Scott Moser
I uploaded open-iscsi_2.0.874-5ubuntu8 which has the debug code
that we were toying with included.  It also includes a fix for
FTBFS bug 1791154.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1788188

Title:
  transient systemd ordering cycle in boot with overlayroot

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  open-iscsi test utilizes overlayroot to boot a cloud-image with root
  filesystem on a read-only iscsi server.

  The /etc/fstab file in the image looks like this:
    LABEL=cloudimg-rootfs   /ext4   defaults0 0
    #LABEL=UEFI /boot/efi   vfatdefaults0 0

  when init takes over from the initramfs, we have

    /proc/cmdline:
  nomodeset iscsi_initiator=maas-enlist
  iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
  iscsi_target_port=3260 iscsi_initiator=maas-enlist
  ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
  
root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
  overlayroot=tmpfs console=ttyS0
  ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

    /etc/fstab:
  # cat /etc/fstab
  #
  #  This fstab is in an overlay. The real one can be found at
  #  /media/root-ro/etc/fstab
  #  The original entry for '/' and other mounts have been updated to be 
placed
  #  under /media/root-ro.
  #  To permanently modify this (or any other file), you should change-root 
into
  #  a writable view of the underlying filesystem using:
  #  sudo overlayroot-chroot
  #
  #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
  #LABEL=UEFI /boot/efi vfat defaults 0 0

   /root/root-ro/etc/fstab:
  LABEL=cloudimg-rootfs   /ext4   defaults0 0
  #LABEL=UEFI /boot/efi   vfatdefaults0 0

   /proc/mounts:
     sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
     proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
     udev /dev devtmpfs 
rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
     devpts /dev/pts devpts 
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
     tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
     /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 
/media/root-ro ext4 ro,relatime 0 0
     tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
     overlayroot / overlay 
ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 0 0

   /proc/1/mountinfo:
     21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
     22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
     23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev 
rw,size=233748k,nr_inodes=58437,mode=755
     24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts 
rw,gid=5,mode=620,ptmxmode=000
     25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs 
rw,size=49288k,mode=755
     26 28 8:1 / /media/root-ro ro,relatime - ext4 
/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
     27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
     28 0 0:24 / / ro,relatime - overlay overlayroot 
ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

  overlayroot's scripts/init-bottom/overlayroot script [1] inherits a
  read-only mount of block device on mount point '$ROOTMNT'
  (ROOTMNT=/root). In order to set up the overlayroot, it does the
  following:

   mkdir /media/root-ro /media/root-rw # in the initramfs
   mount -t tmpfs tmpfs-root /media/root-rw
   mkdir /media/root-rw/overlay-workdir/_
   mount --move $ROOTMNT /media/root-ro
   mount -t overlay -o 
lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 overlayroot /root
   mkdir $ROOTMNT/media/root-ro
   mkdir $ROOTMNT/media/root-rw
   mount --move /media/root-ro "${ROOTMNT}/media/root-ro"
   mount --move /media/root-rw "${ROOTMNT}/media/root-rw"

   # then, if 'ro' on the command line, it mounts /root read-only.
   mount -o remount,ro $ROOTMNT

  The script then exits, as ROOTMNT is now set up with a read-only mount
  of the overlayroot.  All the mounts it has done have been moved
  under ROOTMNT.

  On failure systemd reports:
     [  104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
-.mount/start
     [  104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on 
media-root\x2dro.mount/start
     [  104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle 
starting with media-root\x2dro.mount/start
     [  104.137591] systemd[1]: Requested transaction contains an unfixable 
cyclic 

[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot

2018-09-05 Thread Scott Moser
I've hit that 'retrigger' button 2 more times and have not seen failures.
I just did again. Don't know what to do. :-(

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1788188

Title:
  transient systemd ordering cycle in boot with overlayroot

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  open-iscsi test utilizes overlayroot to boot a cloud-image with root
  filesystem on a read-only iscsi server.

  The /etc/fstab file in the image looks like this:
    LABEL=cloudimg-rootfs   /ext4   defaults0 0
    #LABEL=UEFI /boot/efi   vfatdefaults0 0

  when init takes over from the initramfs, we have

    /proc/cmdline:
  nomodeset iscsi_initiator=maas-enlist
  iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
  iscsi_target_port=3260 iscsi_initiator=maas-enlist
  ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
  
root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
  overlayroot=tmpfs console=ttyS0
  ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

    /etc/fstab:
  # cat /etc/fstab
  #
  #  This fstab is in an overlay. The real one can be found at
  #  /media/root-ro/etc/fstab
  #  The original entry for '/' and other mounts have been updated to be 
placed
  #  under /media/root-ro.
  #  To permanently modify this (or any other file), you should change-root 
into
  #  a writable view of the underlying filesystem using:
  #  sudo overlayroot-chroot
  #
  #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
  #LABEL=UEFI /boot/efi vfat defaults 0 0

   /root/root-ro/etc/fstab:
  LABEL=cloudimg-rootfs   /ext4   defaults0 0
  #LABEL=UEFI /boot/efi   vfatdefaults0 0

   /proc/mounts:
     sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
     proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
     udev /dev devtmpfs 
rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
     devpts /dev/pts devpts 
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
     tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
     /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 
/media/root-ro ext4 ro,relatime 0 0
     tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
     overlayroot / overlay 
ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 0 0

   /proc/1/mountinfo:
     21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
     22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
     23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev 
rw,size=233748k,nr_inodes=58437,mode=755
     24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts 
rw,gid=5,mode=620,ptmxmode=000
     25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs 
rw,size=49288k,mode=755
     26 28 8:1 / /media/root-ro ro,relatime - ext4 
/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
     27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
     28 0 0:24 / / ro,relatime - overlay overlayroot 
ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

  overlayroot's scripts/init-bottom/overlayroot script [1] inherits a
  read-only mount of block device on mount point '$ROOTMNT'
  (ROOTMNT=/root). In order to set up the overlayroot, it does the
  following:

   mkdir /media/root-ro /media/root-rw # in the initramfs
   mount -t tmpfs tmpfs-root /media/root-rw
   mkdir /media/root-rw/overlay-workdir/_
   mount --move $ROOTMNT /media/root-ro
   mount -t overlay -o 
lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 overlayroot /root
   mkdir $ROOTMNT/media/root-ro
   mkdir $ROOTMNT/media/root-rw
   mount --move /media/root-ro "${ROOTMNT}/media/root-ro"
   mount --move /media/root-rw "${ROOTMNT}/media/root-rw"

   # then, if 'ro' on the command line, it mounts /root read-only.
   mount -o remount,ro $ROOTMNT

  The script then exits, as ROOTMNT is now set up with a read-only mount
  of the overlayroot.  All the mounts it has done have been moved
  under ROOTMNT.

  On failure systemd reports:
     [  104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
-.mount/start
     [  104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on 
media-root\x2dro.mount/start
     [  104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle 
starting with media-root\x2dro.mount/start
     [  104.137591] systemd[1]: Requested transaction contains an unfixable 
cyclic ordering dependency: Resource 

[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot

2018-08-23 Thread  Christian Ehrhardt 
I was pushing this through autopkgtesting on LP-Infra where we see issues with 
that sometimes.
Unfortunately when you need an issue it does not show up, I only have 4 good 
cases:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic-ci-train-ppa-service-3372/cosmic/amd64/o/open-iscsi/20180823_070627_52393@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic-ci-train-ppa-service-3372/cosmic/amd64/o/open-iscsi/20180822_061335_52393@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic-ci-train-ppa-service-3372/cosmic/amd64/o/open-iscsi/20180821_215839_7dcd3@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic-ci-train-ppa-service-3372/cosmic/amd64/o/open-iscsi/20180823_083835_52393@/log.gz

I won't retry further, but if anyone wants:
Retrigger is:
https://autopkgtest.ubuntu.com/request.cgi?release=cosmic=amd64=open-iscsi=open-iscsi%2F2.0.874-5ubuntu8%7Eppa0=ci-train-ppa-service%2Fstable-phone-overlay=ci-train-ppa-service%2F3372
And results after a while will show as updated link on:
https://bileto.ubuntu.com/excuses/3372/cosmic.html

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1788188

Title:
  transient systemd ordering cycle in boot with overlayroot

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  open-iscsi test utilizes overlayroot to boot a cloud-image with root
  filesystem on a read-only iscsi server.

  The /etc/fstab file in the image looks like this:
    LABEL=cloudimg-rootfs   /ext4   defaults0 0
    #LABEL=UEFI /boot/efi   vfatdefaults0 0

  when init takes over from the initramfs, we have

    /proc/cmdline:
  nomodeset iscsi_initiator=maas-enlist
  iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
  iscsi_target_port=3260 iscsi_initiator=maas-enlist
  ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
  
root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
  overlayroot=tmpfs console=ttyS0
  ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

    /etc/fstab:
  # cat /etc/fstab
  #
  #  This fstab is in an overlay. The real one can be found at
  #  /media/root-ro/etc/fstab
  #  The original entry for '/' and other mounts have been updated to be 
placed
  #  under /media/root-ro.
  #  To permanently modify this (or any other file), you should change-root 
into
  #  a writable view of the underlying filesystem using:
  #  sudo overlayroot-chroot
  #
  #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
  #LABEL=UEFI /boot/efi vfat defaults 0 0

   /root/root-ro/etc/fstab:
  LABEL=cloudimg-rootfs   /ext4   defaults0 0
  #LABEL=UEFI /boot/efi   vfatdefaults0 0

   /proc/mounts:
     sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
     proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
     udev /dev devtmpfs 
rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
     devpts /dev/pts devpts 
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
     tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
     /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 
/media/root-ro ext4 ro,relatime 0 0
     tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
     overlayroot / overlay 
ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 0 0

   /proc/1/mountinfo:
     21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
     22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
     23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev 
rw,size=233748k,nr_inodes=58437,mode=755
     24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts 
rw,gid=5,mode=620,ptmxmode=000
     25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs 
rw,size=49288k,mode=755
     26 28 8:1 / /media/root-ro ro,relatime - ext4 
/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
     27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
     28 0 0:24 / / ro,relatime - overlay overlayroot 
ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

  overlayroot's scripts/init-bottom/overlayroot script [1] inherits a
  read-only mount of block device on mount point '$ROOTMNT'
  (ROOTMNT=/root). In order to set up the overlayroot, it does the
  following:

   mkdir /media/root-ro /media/root-rw # in the 

[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot

2018-08-22 Thread Scott Moser
here is a pastebin of boot with systemd debugging.
 http://paste.ubuntu.com/p/vGTRh6WB3B/

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1788188

Title:
  transient systemd ordering cycle in boot with overlayroot

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  open-iscsi test utilizes overlayroot to boot a cloud-image with root
  filesystem on a read-only iscsi server.

  The /etc/fstab file in the image looks like this:
    LABEL=cloudimg-rootfs   /ext4   defaults0 0
    #LABEL=UEFI /boot/efi   vfatdefaults0 0

  when init takes over from the initramfs, we have

    /proc/cmdline:
  nomodeset iscsi_initiator=maas-enlist
  iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
  iscsi_target_port=3260 iscsi_initiator=maas-enlist
  ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
  
root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
  overlayroot=tmpfs console=ttyS0
  ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

    /etc/fstab:
  # cat /etc/fstab
  #
  #  This fstab is in an overlay. The real one can be found at
  #  /media/root-ro/etc/fstab
  #  The original entry for '/' and other mounts have been updated to be 
placed
  #  under /media/root-ro.
  #  To permanently modify this (or any other file), you should change-root 
into
  #  a writable view of the underlying filesystem using:
  #  sudo overlayroot-chroot
  #
  #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
  #LABEL=UEFI /boot/efi vfat defaults 0 0

   /root/root-ro/etc/fstab:
  LABEL=cloudimg-rootfs   /ext4   defaults0 0
  #LABEL=UEFI /boot/efi   vfatdefaults0 0

   /proc/mounts:
     sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
     proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
     udev /dev devtmpfs 
rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
     devpts /dev/pts devpts 
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
     tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
     /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 
/media/root-ro ext4 ro,relatime 0 0
     tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
     overlayroot / overlay 
ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 0 0

   /proc/1/mountinfo:
     21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
     22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
     23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev 
rw,size=233748k,nr_inodes=58437,mode=755
     24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts 
rw,gid=5,mode=620,ptmxmode=000
     25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs 
rw,size=49288k,mode=755
     26 28 8:1 / /media/root-ro ro,relatime - ext4 
/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
     27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
     28 0 0:24 / / ro,relatime - overlay overlayroot 
ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

  overlayroot's scripts/init-bottom/overlayroot script [1] inherits a
  read-only mount of block device on mount point '$ROOTMNT'
  (ROOTMNT=/root). In order to set up the overlayroot, it does the
  following:

   mkdir /media/root-ro /media/root-rw # in the initramfs
   mount -t tmpfs tmpfs-root /media/root-rw
   mkdir /media/root-rw/overlay-workdir/_
   mount --move $ROOTMNT /media/root-ro
   mount -t overlay -o 
lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 overlayroot /root
   mkdir $ROOTMNT/media/root-ro
   mkdir $ROOTMNT/media/root-rw
   mount --move /media/root-ro "${ROOTMNT}/media/root-ro"
   mount --move /media/root-rw "${ROOTMNT}/media/root-rw"

   # then, if 'ro' on the command line, it mounts /root read-only.
   mount -o remount,ro $ROOTMNT

  The script then exits, as ROOTMNT is now set up with a read-only mount
  of the overlayroot.  All the mounts it has done have been moved
  under ROOTMNT.

  On failure systemd reports:
     [  104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
-.mount/start
     [  104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on 
media-root\x2dro.mount/start
     [  104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle 
starting with media-root\x2dro.mount/start
     [  104.137591] systemd[1]: Requested transaction contains an unfixable 
cyclic ordering dependency: Resource deadlock avoided

  On 

[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot

2018-08-22 Thread Scott Moser
I pushed a branch up to
 
https://code.launchpad.net/~smoser/ubuntu/+source/open-iscsi/+git/open-iscsi/+ref/bug/1788188-debug
that Christian and I are trying to get some additional debug information out of 
a failure.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1788188

Title:
  transient systemd ordering cycle in boot with overlayroot

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  open-iscsi test utilizes overlayroot to boot a cloud-image with root
  filesystem on a read-only iscsi server.

  The /etc/fstab file in the image looks like this:
    LABEL=cloudimg-rootfs   /ext4   defaults0 0
    #LABEL=UEFI /boot/efi   vfatdefaults0 0

  when init takes over from the initramfs, we have

    /proc/cmdline:
  nomodeset iscsi_initiator=maas-enlist
  iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
  iscsi_target_port=3260 iscsi_initiator=maas-enlist
  ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
  
root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
  overlayroot=tmpfs console=ttyS0
  ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

    /etc/fstab:
  # cat /etc/fstab
  #
  #  This fstab is in an overlay. The real one can be found at
  #  /media/root-ro/etc/fstab
  #  The original entry for '/' and other mounts have been updated to be 
placed
  #  under /media/root-ro.
  #  To permanently modify this (or any other file), you should change-root 
into
  #  a writable view of the underlying filesystem using:
  #  sudo overlayroot-chroot
  #
  #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
  #LABEL=UEFI /boot/efi vfat defaults 0 0

   /root/root-ro/etc/fstab:
  LABEL=cloudimg-rootfs   /ext4   defaults0 0
  #LABEL=UEFI /boot/efi   vfatdefaults0 0

   /proc/mounts:
     sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
     proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
     udev /dev devtmpfs 
rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
     devpts /dev/pts devpts 
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
     tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
     /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 
/media/root-ro ext4 ro,relatime 0 0
     tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
     overlayroot / overlay 
ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 0 0

   /proc/1/mountinfo:
     21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
     22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
     23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev 
rw,size=233748k,nr_inodes=58437,mode=755
     24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts 
rw,gid=5,mode=620,ptmxmode=000
     25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs 
rw,size=49288k,mode=755
     26 28 8:1 / /media/root-ro ro,relatime - ext4 
/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
     27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
     28 0 0:24 / / ro,relatime - overlay overlayroot 
ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

  overlayroot's scripts/init-bottom/overlayroot script [1] inherits a
  read-only mount of block device on mount point '$ROOTMNT'
  (ROOTMNT=/root). In order to set up the overlayroot, it does the
  following:

   mkdir /media/root-ro /media/root-rw # in the initramfs
   mount -t tmpfs tmpfs-root /media/root-rw
   mkdir /media/root-rw/overlay-workdir/_
   mount --move $ROOTMNT /media/root-ro
   mount -t overlay -o 
lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 overlayroot /root
   mkdir $ROOTMNT/media/root-ro
   mkdir $ROOTMNT/media/root-rw
   mount --move /media/root-ro "${ROOTMNT}/media/root-ro"
   mount --move /media/root-rw "${ROOTMNT}/media/root-rw"

   # then, if 'ro' on the command line, it mounts /root read-only.
   mount -o remount,ro $ROOTMNT

  The script then exits, as ROOTMNT is now set up with a read-only mount
  of the overlayroot.  All the mounts it has done have been moved
  under ROOTMNT.

  On failure systemd reports:
     [  104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
-.mount/start
     [  104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on 
media-root\x2dro.mount/start
     [  104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle 
starting with media-root\x2dro.mount/start
     [  

[Touch-packages] [Bug 1788188] Re: transient systemd ordering cycle in boot with overlayroot

2018-08-21 Thread  Christian Ehrhardt 
While it might seem unimportant "only" hitting some open-iscsi tests we
can't be sure how many cases and how many people are hit by this every
now and then.

So I think this really is more important than it might seem at first.

-- 
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to systemd in Ubuntu.
https://bugs.launchpad.net/bugs/1788188

Title:
  transient systemd ordering cycle in boot with overlayroot

Status in systemd package in Ubuntu:
  Confirmed

Bug description:
  open-iscsi test utilizes overlayroot to boot a cloud-image with root
  filesystem on a read-only iscsi server.

  The /etc/fstab file in the image looks like this:
    LABEL=cloudimg-rootfs   /ext4   defaults0 0
    #LABEL=UEFI /boot/efi   vfatdefaults0 0

  when init takes over from the initramfs, we have

    /proc/cmdline:
  nomodeset iscsi_initiator=maas-enlist
  iscsi_target_name=tgt-boot-test-2abbnj iscsi_target_ip=10.0.12.2
  iscsi_target_port=3260 iscsi_initiator=maas-enlist
  ip=maas-enlist:BOOTIF ro net.ifnames=0 BOOTIF_DEFAULT=eth0
  
root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1
  overlayroot=tmpfs console=ttyS0
  ds=nocloud-net;seedfrom=http://10.0.12.2:32600/ init=/bin/bash

    /etc/fstab:
  # cat /etc/fstab
  #
  #  This fstab is in an overlay. The real one can be found at
  #  /media/root-ro/etc/fstab
  #  The original entry for '/' and other mounts have been updated to be 
placed
  #  under /media/root-ro.
  #  To permanently modify this (or any other file), you should change-root 
into
  #  a writable view of the underlying filesystem using:
  #  sudo overlayroot-chroot
  #
  #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/over0
  #LABEL=UEFI /boot/efi vfat defaults 0 0

   /root/root-ro/etc/fstab:
  LABEL=cloudimg-rootfs   /ext4   defaults0 0
  #LABEL=UEFI /boot/efi   vfatdefaults0 0

   /proc/mounts:
     sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
     proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
     udev /dev devtmpfs 
rw,nosuid,relatime,size=233748k,nr_inodes=58437,mode=755 0 0
     devpts /dev/pts devpts 
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
     tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=49288k,mode=755 0 0
     /dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 
/media/root-ro ext4 ro,relatime 0 0
     tmpfs-root /media/root-rw tmpfs rw,relatime 0 0
     overlayroot / overlay 
ro,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 0 0

   /proc/1/mountinfo:
     21 28 0:20 / /sys rw,nosuid,nodev,noexec,relatime - sysfs sysfs rw
     22 28 0:4 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
     23 28 0:6 / /dev rw,nosuid,relatime - devtmpfs udev 
rw,size=233748k,nr_inodes=58437,mode=755
     24 23 0:21 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts 
rw,gid=5,mode=620,ptmxmode=000
     25 28 0:22 / /run rw,nosuid,noexec,relatime - tmpfs tmpfs 
rw,size=49288k,mode=755
     26 28 8:1 / /media/root-ro ro,relatime - ext4 
/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test-2abbnj-lun-1-part1 ro
     27 28 0:23 / /media/root-rw rw,relatime - tmpfs tmpfs-root rw
     28 0 0:24 / / ro,relatime - overlay overlayroot 
ro,lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_

  overlayroot's scripts/init-bottom/overlayroot script [1] inherits a
  read-only mount of block device on mount point '$ROOTMNT'
  (ROOTMNT=/root). In order to set up the overlayroot, it does the
  following:

   mkdir /media/root-ro /media/root-rw # in the initramfs
   mount -t tmpfs tmpfs-root /media/root-rw
   mkdir /media/root-rw/overlay-workdir/_
   mount --move $ROOTMNT /media/root-ro
   mount -t overlay -o 
lowerdir=/media/root-ro,upperdir=/media/root-rw/overlay,workdir=/media/root-rw/overlay-workdir/_
 overlayroot /root
   mkdir $ROOTMNT/media/root-ro
   mkdir $ROOTMNT/media/root-rw
   mount --move /media/root-ro "${ROOTMNT}/media/root-ro"
   mount --move /media/root-rw "${ROOTMNT}/media/root-rw"

   # then, if 'ro' on the command line, it mounts /root read-only.
   mount -o remount,ro $ROOTMNT

  The script then exits, as ROOTMNT is now set up with a read-only mount
  of the overlayroot.  All the mounts it has done have been moved
  under ROOTMNT.

  On failure systemd reports:
     [  104.098833] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
-.mount/start
     [  104.109897] systemd[1]: media-root\x2dro.mount: Found dependency on 
media-root\x2dro.mount/start
     [  104.121386] systemd[1]: media-root\x2dro.mount: Unable to break cycle 
starting with media-root\x2dro.mount/start
     [