Public bug reported:
Both /etc/init/ephemeral-disk-warning.conf and /usr/sbin/ephemeral-disk-
warning contain code to determine where the ephemeral disk is mounted,
write a warning file, and set that file read-only (0444) and immutable
(+i). In the case where the ephemeral drive is bind-mounted to several
places (e.g. using it as a docker backing storage, etc.) an execution of
one of these scripts will yield /mnt 0444 and immutable, usually
breaking whatever was using the storage space.
We had one customer who had a large number of yarn nodes go bad because
of this. Output from `journalctl -u ephemeral-disk-warning.service`:
-- Logs begin at Thu 2018-03-01 00:08:50 UTC, end at Tue 2018-03-13 22:33:22
UTC. --
Mar 01 00:09:14 wn103-limitl systemd[1]: Starting Write warning to Azure
ephemeral disk...
Mar 01 00:09:14 wn103-limitl systemd[1]: Started Write warning to Azure
ephemeral disk.
Mar 05 06:31:44 wn103-limitl systemd[1]: Stopped Write warning to Azure
ephemeral disk.
Mar 05 06:32:41 wn103-limitl systemd[1]: Starting Write warning to Azure
ephemeral disk...
Mar 05 06:32:41 wn103-limitl ephemeral-disk-warning[95793]:
/usr/sbin/ephemeral-disk-warning: line 7: /mnt
Mar 05 06:32:41 wn103-limitl ephemeral-disk-warning[95793]:
/mnt/docker-tmp/plugins
Mar 05 06:32:41 wn103-limitl ephemeral-disk-warning[95793]:
/mnt/docker-tmp/overlay2/DATALOSS_WARNING_README.txt: No such file or directory
Mar 05 06:32:41 wn103-limitl ephemeral-disk-warning[95793]: chmod: cannot
access '/mnt/docker-tmp/overlay2/DATALOSS_WARNING_README.txt': No such file or
directory
Mar 05 06:32:41 wn103-limitl ephemeral-disk-warning[95793]: chattr: No such
file or directory while trying to stat
/mnt/docker-tmp/overlay2/DATALOSS_WARNING_README.txt
Mar 05 06:32:41 wn103-limitl systemd[1]: Started Write warning to Azure
ephemeral disk.
Which was right after the walinuxagent package was updated to
2.2.21+really2.2.20-0ubuntu1~16.04.1
I was able to repro the problematic lines on my dev box, where I have two
directories bind-mounted to my home dir:
$ dev_resource=$(readlink -f /dev/disk/azure/resource-part1)
$ dev_resource_mp=$(awk '$1==R {print$2}' "R=${dev_resource}" /proc/mounts)
$ echo $dev_resource_mp
/mnt /home/paulmey/packer_work /home/paulmey/packer_cache
$ cat /proc/mounts |grep sdb1
/dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0
/dev/sdb1 /home/paulmey/packer_work ext4 rw,relatime,data=ordered 0 0
/dev/sdb1 /home/paulmey/packer_cache ext4 rw,relatime,data=ordered 0 0
It looks like this script needs to be more specific in determining the
mountpoint, or maybe even make its own temporary mount point for the ephemeral
drive, since /proc/mount does not indicate which mounts are bind mounts (and
what the source directory for that bind mount was).
It looks like LP#1626318 is also an example of this bug.
** Affects: walinuxagent (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1755629
Title:
ephemeral-disk-warning yields /mnt 0444 immutable on package update
when /mnt is bind-mounted
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/walinuxagent/+bug/1755629/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
