** Description changed: + === Begin SRU Template === + [Impact] + In some cases, cloud-init writes entries to /etc/fstab, and on azure it will + even format a disk for mounting and then write the entry for that 'ephemeral' + disk there. + + A supported operation on Azure is to "resize" the system. When you do this + the system is shut down, resized (given larger/faster disks and more CPU) and + then brought back up. In that process, the "ephemeral" disk re-initialized + to its original NTFS format. The designed goal is for cloud-init to recognize + this situation and re-format the disk to ext4. + + The problem is that the mount of that disk happens before cloud-init can + reformat. Thats because the entry in fstab has 'auto' and is automatically + mounted. The end result is that after resize operation the user will be left + with the ephemeral disk mounted at /mnt and having a ntfs filesystem rather + than ext4. + + [Test Case] + The text in comment 3 describes how to recreate by the original reporter. + Another way to do this is to just re-format the ephemeral disk as + ntfs and then reboot. The result *should* be that after reboot it + comes back up and has an ext4 filesystem on it. + + 1.) boot system on azure + (for this, i use https://gist.github.com/smoser/5806147, but you can + use web ui or any other way). + + 2.) unmount the ephemeral disk + $ umount /mnt + + 3.) repartition it so that mkfs.ntfs does less and is faster + This is not strictly necessary, but mkfs.ntfs can take upwards of + 20 minutes. shrinking /dev/sdb2 to be 200M means it will finish + in < 1 minute. + + $ disk=/dev/disk/cloud/azure_resource + $ part=/dev/disk/cloud/azure_resource-part1 + $ echo "2048,$((2*1024*100)),7" | sudo sfdisk "$disk" + $ time mkfs.ntfs --quick "$part" + + 4.) reboot + 5.) expect that /proc/mounts has /dev/disk/cloud/azure_resource-part1 as ext4 + and that fstab has x-systemd.requires in it. + + $ awk '$2 == "/mnt" { print $0 }' /proc/mounts + /dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0 + + $ awk '$2 == "/mnt" { print $0 }' /etc/fstab + /dev/sdb1 /mnt auto defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig 0 2 + + [Regression Potential] + Regression is unlikely. Likely failure case is just that the problem is not + correctly fixed, and the user ends up with either an NTFS formated disk that + is mounted at /mnt or there is nothing mounted at /mnt. + + === End SRU Template === + After resizing a 16.04 VM on Azure, the VM is presented with a new ephemeral drive (of a different size), which initially is NTFS formatted. Cloud-init tries to format the appropriate partition ext4, but fails because it is mounted. Cloud-init has unmount logic for exactly this case in the get_data call on the Azure data source, but this is never called because fresh cache is found. Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: start: init-network/check-cache: attempting to read from cache [trust] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Reading from /var/lib/cloud/instance/obj.pkl (quiet=False) Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Read 5950 bytes from /var/lib/cloud/instance/obj.pkl Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] stages.py[DEBUG]: restored from cache: DataSourceAzureNet [seed=/dev/sr0] Jun 27 19:07:47 azubuntu1604arm [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/check-cache: SUCCESS: restored from cache: DataSourceAzureNet [seed=/dev/sr0] ... Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1 Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed return codes [0] (shell=False, capture=True) Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.052 seconds Jun 27 19:07:48 azubuntu1604arm [CLOUDINIT] util.py[WARNING]: Failed during filesystem operation#012Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':#012Unexpected error while running command.#012Command: ['/sbin/mkfs.ext4', '/dev/sdb1']#012Exit code: 1#012Reason: -#012Stdout: ''#012Stderr: 'mke2fs 1.42.13 (17-May-2015)\n/dev/sdb1 is mounted; will not make a filesystem here!\n' $ lsb_release -rd Description: Ubuntu 16.04.1 LTS Release: 16.04 $ cat /etc/cloud/build.info build_name: server serial: 20160721 ~$ dpkg -l cloud-init Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-================================-=====================-=====================-===================================================================== ii cloud-init 0.7.7~bzr1256-0ubuntu all Init scripts for cloud instances We're seeing ~100% repro of this bug on resize, where the only success cases are caused by another bug that messes up fstab and prevents mounting of the drive.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1611074 Title: Reformatting of ephemeral drive fails on resize of Azure VM To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1611074/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
