[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Scott Moser]

** Description changed:

   Begin SRU Template 
  [Impact]
  Overlayfs allows a user to mount the root filesystem read-only
  and put an overlay over it.
  It handles mounting of the root filesystem in the initramfs, and updates
  /etc/fstab to account for non-root filesystems and generally make
  /etc/fstab "correct".
  
  systemd can get confused by a /etc/fstab that looks like this:
  
     LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
     /media/root-ro/ / overlay lowerdir=...
  
  It would not recognize that the LABEL=cloudimg-rootfs image *was* mounted.
  This would lead to boot ordering cycles.  The fix that was ultimately
  applied was just to comment out the line for /media/root-ro.
  
  That way systemd wouldnt try to ensure that /media/root-ro got mounted.
  
  The fix that was put in place was just to comment out the '/' (/media/root-ro)
  line in /etc/fstab.  So essentially:
   - LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
   + #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  
  Since that happens in the initramfs before systemd ever generates mount
  options, it has the desired affect.  And /media/root-ro is already mounted
  before systemd starts.
  
  [Test Case]
  ## setup system to build maas images
  $ sudo apt-get install -qy --no-install-recommends bzr
  $ bzr branch lp:maas-images
  $ cd maas-images
  $ ./system-setup
  
  ## download the reference squashfs images.
  $ arch=amd64
  $ burl=http://cloud-images.ubuntu.com/
  # download reference squashfs images
  $ for rel in xenial zesty artful; do
   file=../$rel-server-cloudimg-$arch.squashfs
   [ -f "$file" ] && continue
   wget "$burl/$rel/current/$file" -O "$file.tmp" && mv "$file.tmp" "$file";
    done
  
  ## build maas image output.  This creates a squashfs image and 
kernel/initramfs
  ## files that have the updated packages inside.
  ## It does take a while to run, and lots of cpu and IO.
  ## You can reduce the kernels built by commenting out 'lowlatency' and 'edge'
  ## kernels:
  ##  sed -i -e '/^[ ].*\(lowlatency\|edge\)/s/^/#/' conf/meph-v3.yaml
  # either set 'proposed' or add the repo via M2E_ADD_REPOS
  $ proposed="--proposed"
  $ export M2E_ADD_REPOS=ppa:smoser/cloud-initramfs-tools
  $ for rel in xenial zesty artful; do
    img=../$rel-server-cloudimg-$arch.squashfs
    PATH=$PWD/bin:$PATH ./bin/meph2-build $proposed --config=conf/meph-v3.yaml \
  $arch $rel $(date +"%Y%m%d") $img ../$rel.d/ 2>&1 |
  tee ../$rel.log
    done
  
  ## After running, you will have files like:
  $ find xenial.d/xenial -type f | sort
  xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-initrd
  xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-kernel
  xenial.d/xenial/amd64/20171221/hwe-16.04/generic/boot-initrd
  xenial.d/xenial/amd64/20171221/hwe-16.04/generic/boot-kernel
  xenial.d/xenial/amd64/20171221/squashfs
  xenial.d/xenial/amd64/20171221/squashfs.manifest
  
  ## You can verify in the $rel.log files you have 'Setting up cloud-initramfs-'
  $ grep '^Setting up overlayroot' xenial.log
  Setting up overlayroot (0.27ubuntu1.5~ppa16.04.1) ...
  
  ## Now download the test-boot script, provide it a directory and
  ## the squashfs, kernel, initramfs
  
  $ ./test-boot test-x xenial.d/xenial/amd64/20171221/squashfs \
  xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-kernel \
  xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-initrd
  
  $ r=zesty; ./test-boot work.d $r.d/*/amd64/*/squashfs 
$r.d/*/amd64/*/ga-*/generic/boot-kernel $r.d/*/amd64/*/ga-*/generic/boot-initrd
  Now, do:
    cd work.d
    ./boot
  
  ## Running ./boot will serve the squashfs image over http, and then
  ## boot the kernel and initrd with a kernel command line of
  ##root=http:///squashfs
  ## and
  ##ds=nocloud-net;seedfrom=http://10.0.2.2:/
  ## which will the read the 'user-data' and 'meta-data' files in that dir.
  ##
  ## The result is that you can log in with 'ubuntu' and 'passw0rd'
  
  ## Then collect some information
  # you'll see the 'LABEL=' entry is commented out.
  $ cat /etc/fstab
  # local-fs.target should be fine
  $ systemctl status local-fs.target --no-pager
  # just for record
  $ cat /proc/cmdline
  
  [Regression Potential]
  Not to omany things would depend on the /etc/fstab that exists in
  an system booted with overlayroot.  Something that was reading /etc/fstab
  could be confused by this change.
  
  [Other Info]
  This change is believed to fix bug 1680197 as well, but due to the
  sporadic nature, we are not certain.
  
  The fix applied here is a backport what is currently in bionic to
  each release. in xenial, that means a new binary package
  (cloud-initramfs-updateroot).
  
   End SRU Template 
  
  I had reported this once before in bug 1666573.
  I thought it was fixed there, but apparently not.
  
  The failure shows itself as (controls removed):
    [   80.089654] systemd[1]: 

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Launchpad Bug Tracker]

This bug was fixed in the package cloud-initramfs-tools - 0.27ubuntu1.5

---
cloud-initramfs-tools (0.27ubuntu1.5) xenial; urgency=medium

  * sync with upstream at 0.40ubuntu1
- whitespace changes.
- rooturl: adjust squashfs url matching to not require '.'.
  Previously it would match *.squashfs or *.squash
  Now it will match *squashfs or *squash.  (LP: #1728048)
- updateroot/*, debian/control: new package 'cloud-initramfs-updateroot'
- debian/README.source: update upstream information for move to git.
- overlayroot: change from storing password in
  /run/initramfs/overlayroot.passwd rather than randomly generated
  /run/initramfs/overlayroot.XX
- overlayroot: avoid confusing systemd (LP: #1723183)
   + Mark / as 'noauto' in the newly written /etc/fstab
   + Comment out the read-only line for /.

 -- Scott Moser   Fri, 15 Dec 2017 15:57:35 -0500

** Changed in: cloud-initramfs-tools (Ubuntu Xenial)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1723183

Title:
  transient systemd ordering issue when using overlayroot

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1723183/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Launchpad Bug Tracker]

This bug was fixed in the package cloud-initramfs-tools - 0.39ubuntu1.1

---
cloud-initramfs-tools (0.39ubuntu1.1) artful; urgency=medium

  * sync with upstream at 0.40ubuntu1
- rooturl: adjust squashfs url matching to not require '.'.
  Previously it would match *.squashfs or *.squash
  Now it will match *squashfs or *.squash.  (LP: #1728048)
- overlayroot: avoid confusing systemd by commenting out read-only
  line for /. (LP: #1723183)
- overlayroot: remove unused variable 'rel_root'.

 -- Scott Moser   Fri, 15 Dec 2017 15:54:28 -0500

** Changed in: cloud-initramfs-tools (Ubuntu Artful)
   Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1723183

Title:
  transient systemd ordering issue when using overlayroot

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1723183/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Scott Moser]

** Attachment added: "verify-log.txt: verification of xenial, zesty and artful 
per test case in descriptiopn"
   
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1723183/+attachment/5037313/+files/verify-log.txt

** Tags removed: verification-needed verification-needed-artful 
verification-needed-xenial verification-needed-zesty
** Tags added: verification-done-artful verification-done-xenial 
verification-done-zesty

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1723183

Title:
  transient systemd ordering issue when using overlayroot

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1723183/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Brian Murray]

Hello Scott, or anyone else affected,

Accepted cloud-initramfs-tools into xenial-proposed. The package will
build now and be available at https://launchpad.net/ubuntu/+source
/cloud-initramfs-tools/0.27ubuntu1.5 in a few hours, and then in the
-proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-xenial to verification-done-xenial. If it does not
fix the bug for you, please add a comment stating that, and change the
tag to verification-failed-xenial. In either case, without details of
your testing we will not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: cloud-initramfs-tools (Ubuntu Xenial)
   Status: In Progress => Fix Committed

** Tags added: verification-needed-xenial

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1723183

Title:
  transient systemd ordering issue when using overlayroot

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1723183/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Brian Murray]

Hello Scott, or anyone else affected,

Accepted cloud-initramfs-tools into zesty-proposed. The package will
build now and be available at https://launchpad.net/ubuntu/+source
/cloud-initramfs-tools/0.35ubuntu2.1 in a few hours, and then in the
-proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-zesty to verification-done-zesty. If it does not fix
the bug for you, please add a comment stating that, and change the tag
to verification-failed-zesty. In either case, without details of your
testing we will not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: cloud-initramfs-tools (Ubuntu Zesty)
   Status: In Progress => Fix Committed

** Tags added: verification-needed-zesty

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1723183

Title:
  transient systemd ordering issue when using overlayroot

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1723183/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Brian Murray]

Hello Scott, or anyone else affected,

Accepted cloud-initramfs-tools into artful-proposed. The package will
build now and be available at https://launchpad.net/ubuntu/+source
/cloud-initramfs-tools/0.39ubuntu1.1 in a few hours, and then in the
-proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested and change the tag from
verification-needed-artful to verification-done-artful. If it does not
fix the bug for you, please add a comment stating that, and change the
tag to verification-failed-artful. In either case, without details of
your testing we will not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Changed in: cloud-initramfs-tools (Ubuntu Artful)
   Status: In Progress => Fix Committed

** Tags added: verification-needed verification-needed-artful

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1723183

Title:
  transient systemd ordering issue when using overlayroot

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1723183/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Scott Moser]

** Description changed:

   Begin SRU Template 
  [Impact]
  Overlayfs allows a user to mount the root filesystem read-only
  and put an overlay over it.
  It handles mounting of the root filesystem in the initramfs, and updates
  /etc/fstab to account for non-root filesystems and generally make
  /etc/fstab "correct".
  
  systemd can get confused by a /etc/fstab that looks like this:
  
     LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
     /media/root-ro/ / overlay lowerdir=...
  
  It would not recognize that the LABEL=cloudimg-rootfs image *was* mounted.
  This would lead to boot ordering cycles.  The fix that was ultimately
  applied was just to comment out the line for /media/root-ro.
  
  That way systemd wouldnt try to ensure that /media/root-ro got mounted.
  
  The fix that was put in place was just to comment out the '/' (/media/root-ro)
  line in /etc/fstab.  So essentially:
   - LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
   + #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
  
  Since that happens in the initramfs before systemd ever generates mount
  options, it has the desired affect.  And /media/root-ro is already mounted
  before systemd starts.
  
  [Test Case]
- TBD
+ ## setup system to build maas images
+ $ sudo apt-get install -qy --no-install-recommends bzr
+ $ bzr branch lp:maas-images
+ $ cd maas-images
+ $ ./system-setup
+ 
+ ## download the reference squashfs images.
+ $ arch=amd64
+ $ burl=http://cloud-images.ubuntu.com/
+ # download reference squashfs images
+ $ for rel in xenial zesty artful; do
+  file=../$rel-server-cloudimg-$arch.squashfs
+  [ -f "$file" ] && continue
+  wget "$burl/$rel/current/$file" -O "$file.tmp" && mv "$file.tmp" "$file";
+   done
+ 
+ ## build maas image output.  This creates a squashfs image and 
kernel/initramfs
+ ## files that have the updated packages inside.
+ ## It does take a while to run, and lots of cpu and IO.
+ ## You can reduce the kernels built by commenting out 'lowlatency' and 'edge'
+ ## kernels:
+ ##  sed -i -e '/^[ ].*\(lowlatency\|edge\)/s/^/#/' conf/meph-v3.yaml
+ # either set 'proposed' or add the repo via M2E_ADD_REPOS
+ $ proposed="--proposed"
+ $ export M2E_ADD_REPOS=ppa:smoser/cloud-initramfs-tools
+ $ for rel in xenial zesty artful; do
+   img=../$rel-server-cloudimg-$arch.squashfs
+   PATH=$PWD/bin:$PATH ./bin/meph2-build $proposed --config=conf/meph-v3.yaml \
+ $arch $rel $(date +"%Y%m%d") $img ../$rel.d/ 2>&1 |
+ tee ../$rel.log
+   done
+ 
+ ## After running, you will have files like:
+ $ find xenial.d/xenial -type f | sort
+ xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-initrd
+ xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-kernel
+ xenial.d/xenial/amd64/20171221/hwe-16.04/generic/boot-initrd
+ xenial.d/xenial/amd64/20171221/hwe-16.04/generic/boot-kernel
+ xenial.d/xenial/amd64/20171221/squashfs
+ xenial.d/xenial/amd64/20171221/squashfs.manifest
+ 
+ ## You can verify in the $rel.log files you have 'Setting up cloud-initramfs-'
+ $ grep '^Setting up overlayroot' xenial.log
+ Setting up overlayroot (0.27ubuntu1.5~ppa16.04.1) ...
+ 
+ 
+ ## Now download the test-boot script, provide it a directory and
+ ## the squashfs, kernel, initramfs
+ 
+ $ ./test-boot test-x xenial.d/xenial/amd64/20171221/squashfs \
+ xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-kernel \
+ xenial.d/xenial/amd64/20171221/ga-16.04/generic/boot-initrd
+ 
+ $ r=zesty; ./test-boot work.d $r.d/*/amd64/*/squashfs 
$r.d/*/amd64/*/ga-*/generic/boot-kernel $r.d/*/amd64/*/ga-*/generic/boot-initrd
+ Now, do:
+   cd work.d
+   ./boot
+ 
+ ## Running ./boot will serve the squashfs image over http, and then
+ ## boot the kernel and initrd with a kernel command line of
+ ##root=http:///squashfs
+ ## and
+ ##ds=nocloud-net;seedfrom=http://10.0.2.2:/
+ ## which will the read the 'user-data' and 'meta-data' files in that dir.
+ ##
+ ## The result is that you can log in with 'ubuntu' and 'passw0rd'
+ 
+ ## Then collect some information
+ # you'll see the 'LABEL=' entry is commented out.
+ $ cat /etc/fstab
+ # local-fs.target should be fine
+ $ systemctl status local-fs.target --no-pager
+ # just for record
+ $ cat /proc/cmdline
  
  [Regression Potential]
  Not to omany things would depend on the /etc/fstab that exists in
  an system booted with overlayroot.  Something that was reading /etc/fstab
  could be confused by this change.
  
  [Other Info]
  This change is believed to fix bug 1680197 as well, but due to the
  sporadic nature, we are not certain.
  
   End SRU Template 
  
  I had reported this once before in bug 1666573.
  I thought it was fixed there, but apparently not.
  
  The failure shows itself as (controls removed):
    [   80.089654] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
media-root\x2dro.mount/start
    [   80.099474] systemd[1]: media-root\x2dro.mount: Found dependency on 

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Scott Moser]

** Description changed:

+  Begin SRU Template 
+ [Impact]
+ Overlayfs allows a user to mount the root filesystem read-only
+ and put an overlay over it.
+ It handles mounting of the root filesystem in the initramfs, and updates
+ /etc/fstab to account for non-root filesystems and generally make
+ /etc/fstab "correct".
+ 
+ systemd can get confused by a /etc/fstab that looks like this:
+ 
+    LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
+    /media/root-ro/ / overlay lowerdir=...
+ 
+ It would not recognize that the LABEL=cloudimg-rootfs image *was* mounted.
+ This would lead to boot ordering cycles.  The fix that was ultimately
+ applied was just to comment out the line for /media/root-ro.
+ 
+ That way systemd wouldnt try to ensure that /media/root-ro got mounted.
+ 
+ [Test Case]
+ TBD
+ 
+ [Regression Potential]
+ Not to omany things would depend on the /etc/fstab that exists in
+ an system booted with overlayroot.  Something that was reading /etc/fstab
+ could be confused by this change.
+ 
+ [Other Info]
+ This change is believed to fix bug 1680197 as well, but due to the
+ sporadic nature, we are not certain.
+ 
+  End SRU Template 
+ 
+ 
  I had reported this once before in bug 1666573.
  I thought it was fixed there, but apparently not.
  
  The failure shows itself as (controls removed):
    [   80.089654] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
media-root\x2dro.mount/start
    [   80.099474] systemd[1]: media-root\x2dro.mount: Found dependency on 
-.mount/start
    [   80.108348] systemd[1]: media-root\x2dro.mount: Found dependency on 
media-root\x2dro.mount/start
    [   80.117231] systemd[1]: Unable to break cycle
    [   80.132286] systemd[1]: Requested transaction contains an unfixable 
cyclic ordering dependency: Resource deadlock avoided
    [   80.180707] systemd[1]: Unnecessary job for VIRTUAL-DISK cloudimg-rootfs 
was removed.
  ...
   [ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
   [DEPEND] Dependency failed for /boot/efi.
   [DEPEND] Dependency failed for Local File Systems.
  
  The /etc/fstab as read from / once we've left the initramfs
  looks like:
    #
    #  This fstab is in an overlay. The real one can be found at
    #  /media/root-ro/etc/fstab
    #  The original entry for '/' and other mounts have been updated to be 
placed
    #  under /media/root-ro.
    #  To permanently modify this (or any other file), you should change-root 
into
    #  a writable view of the underlying filesystem using:
    #  sudo overlayroot-chroot
    #
    LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
    /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_
 0 0
    LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
  
  Notes:
   * I do not believe this is strictly tied to open-iscsi, but the
     test that we see it failing in is the open-iscsi test.
  
  Related bugs:
   * bug 1666573: transient systemd ordering cycle in boot with overlayroot ver 
read-only open-iscsi root
-  * bug 1680197:  Zesty deployments failing sporadically
+  * bug 1680197:  Zesty deployments failing sporadically

** Description changed:

   Begin SRU Template 
  [Impact]
  Overlayfs allows a user to mount the root filesystem read-only
  and put an overlay over it.
  It handles mounting of the root filesystem in the initramfs, and updates
  /etc/fstab to account for non-root filesystems and generally make
  /etc/fstab "correct".
  
  systemd can get confused by a /etc/fstab that looks like this:
  
     LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
     /media/root-ro/ / overlay lowerdir=...
  
  It would not recognize that the LABEL=cloudimg-rootfs image *was* mounted.
  This would lead to boot ordering cycles.  The fix that was ultimately
  applied was just to comment out the line for /media/root-ro.
  
  That way systemd wouldnt try to ensure that /media/root-ro got mounted.
  
+ The fix that was put in place was just to comment out the '/' (/media/root-ro)
+ line in /etc/fstab.  So essentially:
+  - LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
+  + #LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
+ 
+ Since that happens in the initramfs before systemd ever generates mount
+ options, it has the desired affect.  And /media/root-ro is already mounted
+ before systemd starts.
+ 
  [Test Case]
  TBD
  
  [Regression Potential]
  Not to omany things would depend on the /etc/fstab that exists in
  an system booted with overlayroot.  Something that was reading /etc/fstab
  could be confused by this change.
  
  [Other Info]
  This change is believed to fix bug 1680197 as well, but due to the
  sporadic nature, we are not certain.
  
   End SRU Template 
- 
  
  I had reported this once before in bug 1666573.
  I thought it was fixed there, but apparently 

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Scott Moser]

** Also affects: cloud-initramfs-tools (Ubuntu Artful)
   Importance: Undecided
   Status: New

** Also affects: cloud-initramfs-tools (Ubuntu Zesty)
   Importance: Undecided
   Status: New

** Also affects: cloud-initramfs-tools (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Changed in: cloud-initramfs-tools (Ubuntu Xenial)
   Status: New => Confirmed

** Changed in: cloud-initramfs-tools (Ubuntu Zesty)
   Status: New => Confirmed

** Changed in: cloud-initramfs-tools (Ubuntu Artful)
   Status: New => Confirmed

** Changed in: cloud-initramfs-tools (Ubuntu Xenial)
   Importance: Undecided => Medium

** Changed in: cloud-initramfs-tools (Ubuntu Zesty)
   Importance: Undecided => Medium

** Changed in: cloud-initramfs-tools (Ubuntu Artful)
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1723183

Title:
  transient systemd ordering issue when using overlayroot

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1723183/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Scott Moser]

** Description changed:

  I had reported this once before in bug 1666573.
  I thought it was fixed there, but apparently not.
  
  The failure shows itself as (controls removed):
-   [   80.089654] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
media-root\x2dro.mount/start
-   [   80.099474] systemd[1]: media-root\x2dro.mount: Found dependency on 
-.mount/start
-   [   80.108348] systemd[1]: media-root\x2dro.mount: Found dependency on 
media-root\x2dro.mount/start
-   [   80.117231] systemd[1]: Unable to break cycle
-   [   80.132286] systemd[1]: Requested transaction contains an unfixable 
cyclic ordering dependency: Resource deadlock avoided
-   [   80.180707] systemd[1]: Unnecessary job for VIRTUAL-DISK cloudimg-rootfs 
was removed.
+   [   80.089654] systemd[1]: media-root\x2dro.mount: Found ordering cycle on 
media-root\x2dro.mount/start
+   [   80.099474] systemd[1]: media-root\x2dro.mount: Found dependency on 
-.mount/start
+   [   80.108348] systemd[1]: media-root\x2dro.mount: Found dependency on 
media-root\x2dro.mount/start
+   [   80.117231] systemd[1]: Unable to break cycle
+   [   80.132286] systemd[1]: Requested transaction contains an unfixable 
cyclic ordering dependency: Resource deadlock avoided
+   [   80.180707] systemd[1]: Unnecessary job for VIRTUAL-DISK cloudimg-rootfs 
was removed.
  ...
-  [ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
-  [DEPEND] Dependency failed for /boot/efi.
-  [DEPEND] Dependency failed for Local File Systems.
- 
+  [ TIME ] Timed out waiting for device dev-disk-by\x2dlabel-UEFI.device.
+  [DEPEND] Dependency failed for /boot/efi.
+  [DEPEND] Dependency failed for Local File Systems.
  
  The /etc/fstab as read from / once we've left the initramfs
  looks like:
-   #
-   #  This fstab is in an overlay. The real one can be found at
-   #  /media/root-ro/etc/fstab
-   #  The original entry for '/' and other mounts have been updated to be 
placed
-   #  under /media/root-ro.
-   #  To permanently modify this (or any other file), you should change-root 
into
-   #  a writable view of the underlying filesystem using:
-   #  sudo overlayroot-chroot
-   #
-   LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
-   /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_
 0 0
-   LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
+   #
+   #  This fstab is in an overlay. The real one can be found at
+   #  /media/root-ro/etc/fstab
+   #  The original entry for '/' and other mounts have been updated to be 
placed
+   #  under /media/root-ro.
+   #  To permanently modify this (or any other file), you should change-root 
into
+   #  a writable view of the underlying filesystem using:
+   #  sudo overlayroot-chroot
+   #
+   LABEL=cloudimg-rootfs /media/root-ro/ ext4 ro,defaults,noauto 0 0
+   /media/root-ro/ / overlay 
lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay/,workdir=/media/root-rw/overlay-workdir/_
 0 0
+   LABEL=UEFI /boot/efi vfat defaults 0 0 # overlayroot:fs-unsupported
  
  Notes:
-  * I do not believe this is strictly tied to open-iscsi, but the
-test that we see it failing in is the open-iscsi test.
+  * I do not believe this is strictly tied to open-iscsi, but the
+    test that we see it failing in is the open-iscsi test.
  
  Related bugs:
-  * bug 1666573: transient systemd ordering cycle in boot with overlayroot ver 
read-only open-iscsi root
+  * bug 1666573: transient systemd ordering cycle in boot with overlayroot ver 
read-only open-iscsi root
+  * bug 1680197:  Zesty deployments failing sporadically

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1723183

Title:
  transient systemd ordering issue when using overlayroot

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1723183/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Launchpad Bug Tracker]

This bug was fixed in the package cloud-initramfs-tools - 0.40ubuntu1

---
cloud-initramfs-tools (0.40ubuntu1) bionic; urgency=medium

  * rooturl: adjust squashfs url matching to not require '.'.
Previously it would match *.squashfs or *.squash
Now it will match *squashfs or *.squash.
(LP: #1728048)
  * overlayroot: Comment out the read-only line for /.  This hopes to
avoid systemd confusion and transient ordering cycle. (LP: #1723183)

 -- Scott Moser   Fri, 27 Oct 2017 09:44:49 -0400

** Changed in: cloud-initramfs-tools (Ubuntu)
   Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1723183

Title:
  transient systemd ordering issue when using overlayroot

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1723183/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1723183] Re: transient systemd ordering issue when using overlayroot

[Scott Moser]

** Also affects: cloud-initramfs-tools (Ubuntu)
   Importance: Undecided
   Status: New

** Changed in: cloud-initramfs-tools (Ubuntu)
   Status: New => Fix Released

** Changed in: cloud-initramfs-tools (Ubuntu)
   Status: Fix Released => Confirmed

** Changed in: cloud-initramfs-tools (Ubuntu)
   Importance: Undecided => Medium

** Changed in: cloud-initramfs-tools
   Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1723183

Title:
  transient systemd ordering issue when using overlayroot

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1723183/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs