[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2020-01-16 Thread Launchpad Bug Tracker
This bug was fixed in the package linux-hwe - 5.3.0-26.28~18.04.1 --- linux-hwe (5.3.0-26.28~18.04.1) bionic; urgency=medium * bionic/linux-hwe: 5.3.0-26.28~18.04.1 -proposed tracker (LP: #1856849) [ Ubuntu: 5.3.0-26.28 ] * eoan/linux: 5.3.0-26.28 -proposed tracker (LP:

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2020-01-06 Thread Launchpad Bug Tracker
This bug was fixed in the package linux - 5.4.0-9.12 --- linux (5.4.0-9.12) focal; urgency=medium * alsa/hda/realtek: the line-out jack doens't work on a dell AIO (LP: #1855999) - SAUCE: ALSA: hda/realtek - Line-out jack doesn't work on a Dell AIO * scsi: hisi_sas: Check

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2020-01-06 Thread Launchpad Bug Tracker
This bug was fixed in the package linux - 5.0.0-38.41 --- linux (5.0.0-38.41) disco; urgency=medium * disco/linux: 5.0.0-38.41 -proposed tracker (LP: #1854788) * [Regression] Failed to boot disco kernel built from master-next (kernel kernel NULL pointer dereference) (LP:

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2020-01-06 Thread Launchpad Bug Tracker
This bug was fixed in the package linux - 5.3.0-26.28 --- linux (5.3.0-26.28) eoan; urgency=medium * eoan/linux: 5.3.0-26.28 -proposed tracker (LP: #1856807) * nvidia-435 is in eoan, linux-restricted-modules only builds against 430, ubiquity gives me the self-signed modules

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-12-11 Thread Dimitri John Ledkov
** Changed in: linux (Ubuntu Focal) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407 Title: remount of multilower moved pivoted-root overlayfs root,

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-12-08 Thread Colin Ian King
Tested with 5.3.0-25-generic #27-Ubuntu with the regression test and it now works fine. Marking bug as verification-done for eoan ** Tags removed: verification-needed-eoan ** Tags added: verification-done-eoan -- You received this bug notification because you are a member of Ubuntu Bugs, which

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-12-07 Thread Dimitri John Ledkov
The comment in #34 is about 5.3.0-25.27 which is in eoan-proposed. Also ubuntu-kernel-bot must post these message when the kernel is copied into -proposed pocket, and it should state the release series and version, just like all other SRU comments do. -- You received this bug notification

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-12-06 Thread Colin Ian King
Hrm, I can't see the fix in the Ubuntu-5.3.0-24.26 kernel, so I think comment #34 a premature SRU test request. As it stands, I tested Ubuntu-5.3.0-24.26 and the issue still exists, and looking at the source the fix isn't present so that correlates with my test observations. -- You received this

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-12-05 Thread Ubuntu Kernel Bot
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- eoan' to 'verification-done-eoan'. If the problem still exists, change the tag

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-12-04 Thread Colin Ian King
Verified for disco: Run reproducer script with old kernel: 5.0.0-37-generic, results: cat /root-tmp/etc/.pwd.lock cat: /root-tmp/etc/.pwd.lock: Input/output error Run with -proposed kernel: 5.0.0-38-generic cat /root-tmp/etc/.pwd.lock foo Marking as verification-done-disco ** Tags removed:

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-12-03 Thread Ubuntu Kernel Bot
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- disco' to 'verification-done-disco'. If the problem still exists, change the tag

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-28 Thread Stefan Bader
** Changed in: linux (Ubuntu Eoan) Status: New => Fix Committed ** Changed in: linux (Ubuntu Disco) Status: New => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-28 Thread Stefan Bader
** Changed in: linux (Ubuntu Eoan) Importance: Undecided => Critical ** Changed in: linux (Ubuntu Disco) Importance: Undecided => Critical -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-25 Thread Colin Ian King
** Also affects: linux (Ubuntu Focal) Importance: Critical Assignee: Colin Ian King (colin-king) Status: Confirmed ** Also affects: linux-hwe (Ubuntu Focal) Importance: Undecided Status: Invalid ** Also affects: linux (Ubuntu Eoan) Importance: Undecided Status:

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-22 Thread Colin Ian King
** Description changed: + == SRU Justification Disco, Eoan, Focal == + + Multiple squashfs filesystems with overlayfs cause file corruption issues + when modifying zero sized files + + == Fix == + + The current fix is pending in +

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-22 Thread Colin Ian King
I'm doing some testing right now on the current upstream fix, hopefully will SRU this by EOD. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407 Title: remount of multilower moved pivoted-root

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-22 Thread Dimitri John Ledkov
v4 patch is at https://lkml.org/lkml/2019/11/13/799 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407 Title: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-07 Thread Colin Ian King
pr_warn can be removed with a sauce patch, so no worries with that. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407 Title: remount of multilower moved pivoted-root overlayfs root, results in

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-07 Thread Dimitri John Ledkov
pr_warn sounds awkward though Ie. a user will not be able to do anything about it, if that's the only thing they have to mount and boot. Our installer images boot quiet, but this will make it be printed on every boot of Ubuntu installer media. Things that a user didn't cause, and cannot control

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-07 Thread Colin Ian King
been iterating on a fix with upstream: https://lkml.org/lkml/2019/11/7/317 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407 Title: remount of multilower moved pivoted-root overlayfs root,

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Adam Conrad
I didn't mean actually random, and I think clock-based would be entirely fine, but my paranoia about stacking two things with identical superblocks might be just paranoia too. I dunno. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Colin Ian King
When I'm more awake tomorrow I'll send a patch upstream as a suggested fix and see if we can get a good solution on the UUIDs worked out. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407 Title:

Re: [Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Dimitri John Ledkov
I thought all lowerdirs must be read-only, and the first one in the stack will win. Even if there are same ones down the stack it shouldn't matter. But I guess needs testing. I wouldn't want to use random, as that will eat entropy on early boot of Ubuntu Core systems. Do we need a separate

Re: [Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Dimitri John Ledkov
At least it doesn't mangle email notifications! Such that in email code is readable. On Wed, 6 Nov 2019, 18:50 Colin Ian King, <1824...@bugs.launchpad.net> wrote: > Just love the way launchpad mangles pasted code. > > -- > You received this bug notification because you are subscribed to the bug

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Adam Conrad
While there may be arguments for synthesizing UUIDs in various fs drivers, or creating them in the first place in fs-creation tools, I agree that if overlayfs has a hard dependency on UUIDs for uniquely identifying layers, it needs to fill in the gaps where the previous tools/drivers failed to

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Colin Ian King
Just love the way launchpad mangles pasted code. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407 Title: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Colin Ian King
I was thinking of a more generalized overlayfs solution that detects if file systems don't initialize the superblock uuid and overlayfs improvises by generating the internal overlayfs uuid, something like: diff --git a/fs/overlayfs/copy_up.c b/fs/overlayfs/copy_up.c index

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Dimitri John Ledkov
blkid /dev/nvme0n1p1: LABEL="ESP" UUID="B407-D3BC" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="fb8a1f92-fbd6-4ce2-ade4-0a11bdbaf74b" ah, but that UUID is actually fat serial number / volume-id Sets the volume ID of the newly created filesystem; VOLUME-ID is a 32-bit hexadecimal

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Dimitri John Ledkov
There is mkfs-time of the filesystem superblock, which could be converted to a timebased UUID. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407 Title: remount of multilower moved pivoted-root

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Dimitri John Ledkov
I think kernel already generates some faked up UUIDs, i.e. i think there is something for like FAT and DOS partition table, either fs uuid, or like partition table uuid, partition uuid, partition type uuid, some such. They are "shorter" than normal, and less unique, but at least something.

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Colin Ian King
The concern I have is for other file systems that also don't populate the UUID - this seems to be a general problem for overlayfs. Perhaps a UUID can be autogenerated based on the superblock rather than file system specific UUID magic if the UUID is zero. -- You received this bug notification

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Colin Ian King
Adding a uuid into the superblock on squashfs seems to resolve the issue. Since squashfs does not have UUID support, my hack below generates one based on some squashfs superblock metadata that provides a good enough UUID for our purposes. diff --git a/fs/squashfs/super.c b/fs/squashfs/super.c

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-06 Thread Colin Ian King
Comparing the previous debug with 2 squashfs overlayfs lowers with the *same* data on ext4 as the 2 overlayfs lowers we have: [ 56.257691] repro-nosquashf (1038): drop_caches: 3 [ 56.265075] ovl_get_fh: 112 dentry: etc/.pwd.lock name: trusted.overlay.origin [ 56.265077] ovl_get_fh: 115:

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-05 Thread Colin Ian King
4.15: ovl_get_origin_fh detects zero sized files on lower paths and treats these a special zero sized "copied up but origin unknown" magic. [ 25.442916] ovl_check_origin: etc/.pwd.lock 2 [ 25.442918] ovl_get_origin_fh: 104 etc/.pwd.lock [ 25.442919] ovl_get_origin_fh: 107 res=0 [

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-05 Thread Colin Ian King
Replaced read-only squashfs with read-only ext4 partitions and can't reproduce the error. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407 Title: remount of multilower moved pivoted-root

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-05 Thread Colin Ian King
Replaced one of the two squashfs with read-only ext4 partitions and can't reproduce the error. Seems that we need 2 stacked squashfs file systems. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1824407

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-04 Thread Colin Ian King
OK, now managed to get a reproducer script to kick this bug even outside the early install context. Seems like we can force this bug by either remounting OR sync'ing and dropping caches. Attached is the reproducer script. Run as root, we hit the error: cat: /root-tmp/etc/.pwd.lock:

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-04 Thread Colin Ian King
And if we change use a different file: /root-tmp/var/log/ubuntu- advantage.log we get the following error too: [ 24.531406] SQUASHFS error: squashfs_read_data failed to read block 0x89c066e0b540 [ 24.531444] SQUASHFS error: Unable to read metadata cache entry [89c066e0b540] -- You

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-04 Thread Colin Ian King
Hi Dimitri, while debugging this I found the following in setup_unionfs() in scripts/casper: # move the first mount; no head in busybox-initramfs for d in $(mount -t squashfs | cut -d\ -f 3); do mkdir -p "${rootmnt}/rofs" if [ "${UNIONFS}" = unionfs-fuse ]; then

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-04 Thread Colin Ian King
BTW, I can generate the mount move failure with the cut down script as follows (which follows the same mount patterns as the casper script) #!/bin/bash -x mkdir -p /cdrom mount -t iso9660 -o ro,noatime /dev/sr0 /cdrom sleep 1 mkdir -p /cow mount -t tmpfs -o 'rw,noatime,mode=755' tmpfs /cow sleep

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-01 Thread Dimitri John Ledkov
** Description changed: 1) Download focal subiquity pending image, or eoan release image 2) boot, and press ESC and edit boot command line (F6 in bios, e in UEFI) - 3) Before --- insert the following options + 3) After --- insert the following options    break=top debug init=/bin/bash

[Bug 1824407] Re: remount of multilower moved pivoted-root overlayfs root, results in I/O errors on some modified files

2019-11-01 Thread Colin Ian King
** Changed in: linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) ** Changed in: linux-hwe (Ubuntu Bionic) Assignee: (unassigned) => Colin Ian King (colin-king) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to