Please do not go to release with this issue! This is a MAJOR regression compared to 19.10, and considering the amount of pain the previous persistent issue with LTS has caused (just go to reddit, superuser or askubuntu and search for "mounting /cow on /root/cow failed" reports to get an idea of just how many Ubuntu users are being negatively impacted by this), if Ubuntu does value its userbase, it does not want a repeat of the 18.04 persistence fiasco for the next LTS.
I will provide steps on how to replicate the issue below, but first of all, I want to express my puzzlement with regards to unilaterally deciding to introduce a new 'writable' label for persistent partitions. Why???? "writable" provides no hints about persistence, and is way too generic to actually be understood. Unless you are using an optical media, where it's the hardware itself that is preventing write access, tagging a partition as "writeable" yields nothing, since, in the absolute, EVERY partition is of course writeable. So this tells nothing of value to users when it comes to persistence. Why not take a page from Debian here and just use 'persistence' for the label? At least, if people don't understand why their partition is called that way, they can perform a search and get a good overview of what persistent means. But with 'writable', you get nothing of the sort. And seriously, for crying out loud, could you please coordinate with other distros when it comes to persistence. It's bad enough that Ubuntu chose to use a different kernel option ('persistent' vs. Debian's 'persistence') but you had a golden opportunity to bridge a gap by following what Debian was doing (by introducing the new 'persistence' partition label), yet chose once again to disregard the common good and do it your way. Which brings me to the main point, the 'casper-rw' race condition is a MAJOR issue, which is far from being limited to slow machines. I am seeing it consistently happening on all of the UEFI platforms I am testing with, be it relatively slow dual-core based platform from >5 years ago to a recent 6-cores platform bought in January this year. So can you *PLEASE* test on real hardware instead of on VMs? I am confident that if you do test it on real hardware by creating a media in the manner highlighted below, you will consistently see the issue. Below are the exact commands I used to create a UEFI-bootable persistent USB Flash Drive, which you should be able to follow more or less exactly to replicate the problem (tested with the latest focal-desktop-amd64.iso daily build from 2020.04.03): ----------------------------------------------------------------------------------------- root@nano:/# ## Make sure to change the following disk to your USB media root@nano:/# export TARGET_DISK=/dev/sda root@nano:/# ## The following two commands erase the partition tables root@nano:/# dd if=/dev/zero of=$TARGET_DISK bs=512 count=34 34+0 records in 34+0 records out 17408 bytes (17 kB, 17 KiB) copied, 0.00602524 s, 2.9 MB/s root@nano:/# dd if=/dev/zero of=$TARGET_DISK bs=512 count=34 seek=$((`blockdev --getsz $TARGET_DISK` - 34)) 34+0 records in 34+0 records out 17408 bytes (17 kB, 17 KiB) copied, 0.0249334 s, 698 kB/s root@nano:/# gdisk $TARGET_DISK GPT fdisk (gdisk) version 1.0.5 Partition table scan: MBR: not present BSD: not present APM: not present GPT: not present Creating new GPT entries in memory. Command (? for help): n Partition number (1-128, default 1): First sector (34-31277198, default = 2048) or {+-}size{KMGTP}: Last sector (2048-31277198, default = 31277198) or {+-}size{KMGTP}: +10G Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): 0700 Changed type of partition to 'Microsoft basic data' Command (? for help): n Partition number (2-128, default 2): First sector (34-31277198, default = 20973568) or {+-}size{KMGTP}: Last sector (20973568-31277198, default = 31277198) or {+-}size{KMGTP}: Current type is 8300 (Linux filesystem) Hex code or GUID (L to show codes, Enter = 8300): Changed type of partition to 'Linux filesystem' Command (? for help): p Disk /dev/sda: 31277232 sectors, 14.9 GiB Model: Extreme Sector size (logical/physical): 512/512 bytes Disk identifier (GUID): 0E42CF4F-3EB9-420E-A5A1-6AE9E3D746F0 Partition table holds up to 128 entries Main partition table begins at sector 2 and ends at sector 33 First usable sector is 34, last usable sector is 31277198 Partitions will be aligned on 2048-sector boundaries Total free space is 2014 sectors (1007.0 KiB) Number Start (sector) End (sector) Size Code Name 1 2048 20973567 10.0 GiB 0700 Microsoft basic data 2 20973568 31277198 4.9 GiB 8300 Linux filesystem Command (? for help): w Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!! Do you want to proceed? (Y/N): Y OK; writing new GUID partition table (GPT) to /dev/sda. The operation has completed successfully. root@nano:/# mkfs.vfat ${TARGET_DISK}1 mkfs.fat 4.1 (2017-01-24) root@nano:/# mkfs.ext3 -L casper-rw ${TARGET_DISK}2 mke2fs 1.45.6 (20-Mar-2020) Creating filesystem with 1287953 4k blocks and 322560 inodes Filesystem UUID: b8f3305d-17ab-4344-a611-b2817fe8fcb3 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done root@nano:/# mount -o loop /mnt/ssd/focal-desktop-amd64.iso /mnt/iso mount: /mnt/iso: WARNING: device write-protected, mounted read-only. root@nano:/# mount ${TARGET_DISK}1 /mnt/hd root@nano:/# cp -rT /mnt/iso /mnt/hd cp: cannot create symbolic link '/mnt/hd/ubuntu': Operation not permitted cp: cannot create symbolic link '/mnt/hd/dists/stable': Operation not permitted cp: cannot create symbolic link '/mnt/hd/dists/unstable': Operation not permitted root@nano:/# ## The following adds the 'persistent' option to 'grub.cfg' root@nano:/# sed -i 's/\/casper\/vmlinuz\$casper_flavour /\/casper\/vmlinuz\$casper_flavour persistent /g' /mnt/hd/boot/grub/grub.cfg root@nano:/# umount /mnt/hd root@nano:/# umount /mnt/iso root@nano:/# sync ----------------------------------------------------------------------------------------- Then, when you boot the media on a UEFI platform, the persistent partition consistently fails to mount (I believe I saw it actually mount twice in about 20 tryouts, which does indeed seem to indicate a race condition). FYI, this is what you may see in Ubuntu when the persistent media above fails to activate persistence: ----------------------------------------------------------------------------------------- To run a command as administrator (user "root"), use "sudo <command>". See "man sudo_root" for details. ubuntu@ubuntu:~$ df -h Filesystem Size Used Avail Use% Mounted on udev 7.7G 0 7.7G 0% /dev tmpfs 1.6G 1.9M 1.6G 1% /run /dev/sda1 10G 2.7G 7.4G 27% /cdrom /dev/loop0 2.1G 2.1G 0 100% /rofs /cow 7.7G 129M 7.6G 2% / tmpfs 7.7G 0 7.7G 0% /dev/shm tmpfs 5.0M 4.0K 5.0M 1% /run/lock tmpfs 7.7G 0 7.7G 0% /sys/fs/cgroup tmpfs 7.7G 0 7.7G 0% /tmp tmpfs 1.6G 44K 1.6G 1% /run/user/999 /dev/sda2 4.8G 10M 4.6G 1% /media/ubuntu/casper-rw /dev/loop1 94M 94M 0 100% /snap/core/8935 ubuntu@ubuntu:~$ cat /var/log/casper.log mount: mounting /cow on /root/cow failed: No such file or directory passwd: password expiry information changed. dbus-daemon[1069]: [session uid=999 pid=1069] Activating service name='org.gtk.vfs.Daemon' requested by ':1.0' (uid=999 pid=1070 comm="" label="unconfined") dbus-daemon[1069]: [session uid=999 pid=1069] Successfully activated service 'org.gtk.vfs.Daemon' dbus-daemon[1069]: [session uid=999 pid=1069] Activating service name='org.gtk.vfs.Metadata' requested by ':1.0' (uid=999 pid=1070 comm="" label="unconfined") fuse: device not found, try 'modprobe fuse' first dbus-daemon[1069]: [session uid=999 pid=1069] Successfully activated service 'org.gtk.vfs.Metadata' A connection to the bus can't be made Using CD-ROM mount point /cdrom/ Identifying... [e4f884762ec30c968a13f203cdef4b44-2] Scanning disc for index files... Found 2 package indexes, 0 source indexes, 0 translation indexes and 1 signatures Found label 'Ubuntu 20.04 LTS _Focal Fossa_ - Beta amd64 (20200402)' This disc is called: 'Ubuntu 20.04 LTS _Focal Fossa_ - Beta amd64 (20200402)' Copying package lists...gpgv: Signature made Fri Apr 3 07:35:50 2020 UTC gpgv: using RSA key D94AA3F0EFE21092 gpgv: Good signature from "Ubuntu CD Image Automatic Signing Key (2012) <cdim...@ubuntu.com>" Reading Package Indexes... Done Writing new source list Source list entries for this disc are: deb cdrom:[Ubuntu 20.04 LTS _Focal Fossa_ - Beta amd64 (20200402)]/ focal main restricted Repeat this process for the rest of the CDs in your set. ubuntu@ubuntu:~$ cat /var/log/boot.log cat: /var/log/boot.log: Permission denied ubuntu@ubuntu:~$ sudo cat /var/log/boot.log ------------ Sat Apr 04 14:38:24 UTC 2020 ------------ ln: /tmp/mountroot-fail-hooks.d//scripts/init-premount/lvm2: No such file or directory mount: mounting /cow on /root/cow failed: No such file or directory passwd: password expiry information changed. dbus-daemon[1069]: [session uid=999 pid=1069] Activating service name='org.gtk.vfs.Daemon' requested by ':1.0' (uid=999 pid=1070 comm="" label="unconfined") dbus-daemon[1069]: [session uid=999 pid=1069] Successfully activated service 'org.gtk.vfs.Daemon' dbus-daemon[1069]: [session uid=999 pid=1069] Activating service name='org.gtk.vfs.Metadata' requested by ':1.0' (uid=999 pid=1070 comm="" label="unconfined") fuse: device not found, try 'modprobe fuse' first dbus-daemon[1069]: [session uid=999 pid=1069] Successfully activated service 'org.gtk.vfs.Metadata' A connection to the bus can't be made Using CD-ROM mount point /cdrom/ Identifying... [e4f884762ec30c968a13f203cdef4b44-2] Scanning disc for index files... Found 2 package indexes, 0 source indexes, 0 translation indexes and 1 signatures Found label 'Ubuntu 20.04 LTS _Focal Fossa_ - Beta amd64 (20200402)' This disc is called: 'Ubuntu 20.04 LTS _Focal Fossa_ - Beta amd64 (20200402)' Copying package lists...gpgv: Signature made Fri Apr 3 07:35:50 2020 UTC gpgv: using RSA key D94AA3F0EFE21092 gpgv: Good signature from "Ubuntu CD Image Automatic Signing Key (2012) <cdim...@ubuntu.com>" Reading Package Indexes... Done Writing new source list Source list entries for this disc are: deb cdrom:[Ubuntu 20.04 LTS _Focal Fossa_ - Beta amd64 (20200402)]/ focal main restricted Repeat this process for the rest of the CDs in your set. [FAILED] Failed unmounting /cdrom. [ OK ] Finished Tell Plymouth To Write Out Runtime Data. [ OK ] Finished Create Volatile Files and Directories. Starting Network Name Resolution... Starting Network Time Synchronization... Starting Update UTMP about System Boot/Shutdown... [ OK ] Finished Update UTMP about System Boot/Shutdown. [ OK ] Started Network Name Resolution. [ OK ] Started Network Time Synchronization. [ OK ] Reached target Host and Network Name Lookups. [ OK ] Reached target System Initialization. [ OK ] Started ACPI Events Check. [ OK ] Started CUPS Scheduler. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Clean up old snapshots to free space. [ OK ] Reached target Paths. [ OK ] Reached target System Time Set. [ OK ] Reached target System Time Synchronized. [ OK ] Started Trigger anacron every hour. [ OK ] Started Daily apt download activities. [ OK ] Started Daily apt upgrade and clean activities. [ OK ] Started Periodic ext4 Online Metadata Check for All Filesystems. [ OK ] Started Discard unused blocks once a week. [ OK ] Started Refresh fwupd metadata regularly. [ OK ] Started Daily rotation of log files. [ OK ] Started Daily man-db regeneration. [ OK ] Started Message of the Day. [ OK ] Reached target Timers. [ OK ] Listening on ACPID Listen Socket. [ OK ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket. [ OK ] Listening on CUPS Scheduler. [ OK ] Listening on D-Bus System Message Bus Socket. Starting Socket activation for snappy daemon. [ OK ] Listening on UUID daemon activation socket. [ OK ] Listening on Socker activation for zsys daemon. [ OK ] Listening on Socket activation for snappy daemon. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Accounts Service... [ OK ] Started ACPI event daemon. [ OK ] Started Run anacron jobs. Starting LSB: automatic crash report generation... Starting Avahi mDNS/DNS-SD Stack... Starting Bluetooth service... [ OK ] Started Regular background program processing daemon. [ OK ] Started CUPS Scheduler. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Save initial kernel messages after boot. Starting Remove Stale Online ext4 Metadata Check Snapshots... [ OK ] Reached target Login Prompts. Starting Detect the available GPUs and deal with any system changes... Starting LSB: Record successful boot for GRUB... Starting GRUB failed boot detection... [ OK ] Started irqbalance daemon. Starting Dispatcher daemon for systemd-networkd... [ OK ] Started Set the CPU Frequency Scaling governor. Starting Authorization Manager... Starting Restore /etc/resolv.conf if the system crashed before the ppp link was shut down... Starting System Logging Service... Starting Secure Boot updates for DB and DBX... Starting Snap Daemon... Starting Switcheroo Control Proxy service... Starting Login Service... Starting Thermal Daemon Service... Starting Ubuntu live CD installer... Starting Disk Manager... Starting WPA supplicant... Starting ZFS file system shares... [ OK ] Started ZFS Event Daemon (zed). [ OK ] Finished Restore /etc/resolv.conf if the system crashed before the ppp link was shut down. [ OK ] Finished GRUB failed boot detection. [ OK ] Finished ZFS file system shares. [ OK ] Reached target ZFS startup target. [ OK ] Finished Ubuntu live CD installer. [ OK ] Finished Secure Boot updates for DB and DBX. [ OK ] Finished Remove Stale Online ext4 Metadata Check Snapshots. [ OK ] Started LSB: Record successful boot for GRUB. [ OK ] Started LSB: automatic crash report generation. [ OK ] Started System Logging Service. [ OK ] Finished Detect the available GPUs and deal with any system changes. [ OK ] Started Bluetooth service. [ OK ] Started Avahi mDNS/DNS-SD Stack. [ OK ] Reached target Bluetooth. [ OK ] Started Make remote CUPS printers available locally. [ OK ] Started Thermal Daemon Service. [ OK ] Started WPA supplicant. [ OK ] Started Switcheroo Control Proxy service. Starting Save/Restore Sound Card State... [ OK ] Finished Save/Restore Sound Card State. [ OK ] Reached target Sound Card. Starting Hostname Service... [ OK ] Started Authorization Manager. Starting Modem Manager... Starting Thunderbolt system service... [ OK ] Started Accounts Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Network Manager Wait Online... Starting OpenVPN service... Starting Permit User Sessions... [ OK ] Finished OpenVPN service. [ OK ] Finished Permit User Sessions. Starting GNOME Display Manager... Starting Hold until boot process finishes up... [ OK ] Started GNOME Display Manager. [ OK ] Started Modem Manager. [ OK ] Started Disk Manager. ubuntu@ubuntu:~$ mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,nosuid,noexec,relatime,size=8003244k,nr_inodes=2000811,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1610228k,mode=755) /dev/sda1 on /cdrom type vfat (ro,noatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro) /dev/loop0 on /rofs type squashfs (ro,noatime) /cow on / type overlay (rw,relatime,lowerdir=/filesystem.squashfs,upperdir=/cow/upper,workdir=/cow/work) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21356) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime) tmpfs on /run/user/999 type tmpfs (rw,nosuid,nodev,relatime,size=1610224k,mode=700,uid=999,gid=999) gvfsd-fuse on /run/user/999/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=999,group_id=999) /dev/fuse on /run/user/999/doc type fuse (rw,nosuid,nodev,relatime,user_id=999,group_id=999) /dev/sda2 on /media/ubuntu/casper-rw type ext3 (rw,nosuid,nodev,relatime,uhelper=udisks2) /var/lib/snapd/snaps/core_8935.snap on /snap/core/8935 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/core18_1705.snap on /snap/core18/1705 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/gnome-3-34-1804_21.snap on /snap/gnome-3-34-1804/21 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/gtk-common-themes_1474.snap on /snap/gtk-common-themes/1474 type squashfs (ro,nodev,relatime,x-gdu.hide) /var/lib/snapd/snaps/snap-store_336.snap on /snap/snap-store/336 type squashfs (ro,nodev,relatime,x-gdu.hide) ubuntu@ubuntu:~$ ----------------------------------------------------------------------------------------- If you create the media using 'writable' instead of 'casper-rw', or relabel the existing partition to 'writable', then persistence appears to be set up properly (though you still get a "mount: mounting /cow on /root/cow failed: No such file or directory" error), and while I understand that some may see this as a suitable workaround, this is actually problematic because: - I am the developer of Rufus, which is currently the application Ubuntu officially recommends to create bootable Ubuntu media on Windows. - Because of the above, I spent a lot of time trying to ensure that I could let Windows users enable persistence when creating Ubuntu bootable media, using pretty much the method documented above. Thanks to Michael's effort, who managed to fix the major persistence bug that existed in #1489855, this lead to people using Ubuntu 19.10 happily creating persistent media. - Even as this resulted in a lot of flak ("I'm trying to use Rufus 18.04 and persistence doesn't work -- Fix your application!!"), I took the party that the Ubuntu maintainers would recognize the severity of having broken persistence, and therefore: a) Retroffit the #1489855 bugfix into 18.04 (still hasn't happened as of 18.04.4) b) Ensure that future versions of Ubuntu would be tested for regressions for the type of persistent media creation I have detailed above, because it is one of the simplest way to manually create a persistent bootable UEFI media. - It is problematic to add version detection when there exist many Ubuntu derivatives that would need to be differentiated (using what criteria???) to decide whether persistence should be available as an option or not, as well as what label should be used for the persistent partition (how the heck am I supposed to tell if a future derivative such a Linux Mint is using a version of the casper scripts that can use 'writable' as a label?) As such, I decided not add blanket Ubuntu version detection in Rufus to decide whether creation of persistent partitions should be enabled or disabled in the UI, because I did fully expect the 'casper-rw' persitent partition situation to improve over time. This issue however demonstrates that, as opposed to what I had hoped, Ubuntu is not serioulsy committed to testing persistence and/or ensuring that regressions don't happen, which is very problematic, not only for developers of applications that aime at creating persistent bootable media, but for Ubuntu users as a whole, who, after having unable for YEARS to be able to create persistent media with 18.04 LTS, are going to find, again, that trying to enable persistence with 20.04 LTS is a major game of russian roulette, replete with tons of contradictory information ("You should use 'writeable'" "But all the guides I see tell me I need to use 'casper-rw'!"). Sorry to sound flippant here, but Ubuntu REALLY seems to have dropped the ball on that one, and I currently have no idea what I'm going to be able to do in Rufus to try to salvage the current situation, without having to keep persistence broken for either 19.10 or 20.04 users... So can you PLEASE get your act together and either atempt to fix this before the 20.04 release, or just revert to using the 19.10 scripts whilst dropping the idea of introducing an incomprehensible new 'writable' persistent label out of the blue? Thank you. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1863672 Title: The 'new' persistent live method starting in 19.10 no longer works To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1863672/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs