Public bug reported: Both /etc/init/ephemeral-disk-warning.conf and /usr/sbin/ephemeral-disk- warning contain code to determine where the ephemeral disk is mounted, write a warning file, and set that file read-only (0444) and immutable (+i). In the case where the ephemeral drive is bind-mounted to several places (e.g. using it as a docker backing storage, etc.) an execution of one of these scripts will yield /mnt 0444 and immutable, usually breaking whatever was using the storage space.
We had one customer who had a large number of yarn nodes go bad because of this. Output from `journalctl -u ephemeral-disk-warning.service`: -- Logs begin at Thu 2018-03-01 00:08:50 UTC, end at Tue 2018-03-13 22:33:22 UTC. -- Mar 01 00:09:14 wn103-limitl systemd[1]: Starting Write warning to Azure ephemeral disk... Mar 01 00:09:14 wn103-limitl systemd[1]: Started Write warning to Azure ephemeral disk. Mar 05 06:31:44 wn103-limitl systemd[1]: Stopped Write warning to Azure ephemeral disk. Mar 05 06:32:41 wn103-limitl systemd[1]: Starting Write warning to Azure ephemeral disk... Mar 05 06:32:41 wn103-limitl ephemeral-disk-warning[95793]: /usr/sbin/ephemeral-disk-warning: line 7: /mnt Mar 05 06:32:41 wn103-limitl ephemeral-disk-warning[95793]: /mnt/docker-tmp/plugins Mar 05 06:32:41 wn103-limitl ephemeral-disk-warning[95793]: /mnt/docker-tmp/overlay2/DATALOSS_WARNING_README.txt: No such file or directory Mar 05 06:32:41 wn103-limitl ephemeral-disk-warning[95793]: chmod: cannot access '/mnt/docker-tmp/overlay2/DATALOSS_WARNING_README.txt': No such file or directory Mar 05 06:32:41 wn103-limitl ephemeral-disk-warning[95793]: chattr: No such file or directory while trying to stat /mnt/docker-tmp/overlay2/DATALOSS_WARNING_README.txt Mar 05 06:32:41 wn103-limitl systemd[1]: Started Write warning to Azure ephemeral disk. Which was right after the walinuxagent package was updated to 2.2.21+really2.2.20-0ubuntu1~16.04.1 I was able to repro the problematic lines on my dev box, where I have two directories bind-mounted to my home dir: $ dev_resource=$(readlink -f /dev/disk/azure/resource-part1) $ dev_resource_mp=$(awk '$1==R {print$2}' "R=${dev_resource}" /proc/mounts) $ echo $dev_resource_mp /mnt /home/paulmey/packer_work /home/paulmey/packer_cache $ cat /proc/mounts |grep sdb1 /dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0 /dev/sdb1 /home/paulmey/packer_work ext4 rw,relatime,data=ordered 0 0 /dev/sdb1 /home/paulmey/packer_cache ext4 rw,relatime,data=ordered 0 0 It looks like this script needs to be more specific in determining the mountpoint, or maybe even make its own temporary mount point for the ephemeral drive, since /proc/mount does not indicate which mounts are bind mounts (and what the source directory for that bind mount was). It looks like LP#1626318 is also an example of this bug. ** Affects: walinuxagent (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1755629 Title: ephemeral-disk-warning yields /mnt 0444 immutable on package update when /mnt is bind-mounted To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/walinuxagent/+bug/1755629/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs