** Merge proposal linked:
https://code.launchpad.net/~simpoir/livecd-rootfs/+git/livecd-rootfs/+merge/465193
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to util-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2045586
I tested the flock-based solution with some of the CPC pipelines in
jammy and saw consistently clean builds (30 successful images built
yesterday). Thank you very much for everyone's hard work debugging and
fixing this race condition!
--
You received this bug notification because you are a
** Merge proposal linked:
https://code.launchpad.net/~ubuntu-core-dev/livecd-rootfs/+git/livecd-rootfs/+merge/460810
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to util-linux in Ubuntu.
This bug was fixed in the package livecd-rootfs - 2.765.39
---
livecd-rootfs (2.765.39) jammy; urgency=medium
[ dann frazier ]
* Use flock to avoid races with systemd-udevd that cause loop device
partitions to briefly disappear. (LP: #2045586)
-- Michael Hudson-Doyle Mon,
Releasing early as we're in the middle of a point-release. Autopkgtests
look good, the actual testing will happen on the RCs.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to util-linux in Ubuntu.
Hello Steve, or anyone else affected,
Accepted livecd-rootfs into jammy-proposed. The package will build now
and be available at https://launchpad.net/ubuntu/+source/livecd-
rootfs/2.765.39 in a few hours, and then in the -proposed repository.
Please help us by testing this new package. See
** Description changed:
- In mantic, we migrated livecd-rootfs to use losetup -P instead of
- kpartx, with the expectation that this would give us a reliable, race-
- free way of loop-mounting partitions from a disk image during image
- build.
+ [impact]
+ In mantic, we migrated livecd-rootfs to
** Merge proposal linked:
https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/460732
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to util-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2045586
** Merge proposal linked:
https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/460729
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to util-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2045586
Here is a backport of the partial fix to jammy
https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-
rootfs/+merge/460729
I'm not sure I am best placed to update this bug description to match
the SRU template though -- I'm not sure which builds are being affected
in practice and so
I'm glad to hear you are seeing an improvement! The current
implementation is still racy as mentioned in Comment #22. I did a search
of the logs I downloaded, and I believe this one shows that the
mount_partition() race isn't just theoretical:
https://autopkgtest.ubuntu.com/results/autopkgtest-
I've been seeing this bug as far back as losetup is used (jammy) and the
previously merged fix has a significant improvement on successful builds
(0/3 succeeded without the patch applied vs 5/5 succeeded with the patch
applied this week) in jammy. Is someone already looking to SRU that
patch back
While the above MP is now merged, I still see additional potential races
in the code. For example, anything calling mount_partition() for the
first partition, and also maybe bug 2030771. I don't see a good way to
solve that w/o `udevadm lock` because these functions don't currently
know what the
** Also affects: util-linux (Ubuntu Mantic)
Importance: Undecided
Status: New
** Also affects: livecd-rootfs (Ubuntu Mantic)
Importance: Undecided
Status: New
** Also affects: linux (Ubuntu Mantic)
Importance: Undecided
Status: New
** Also affects: util-linux
The test passed w/ flock as well:
https://autopkgtest.ubuntu.com/results/autopkgtest-noble-dannf-loop/noble/amd64/l/livecd-rootfs/20240131_104633_df869@/log.gz
274 successful iterations before the timeout killed it.
I've updated the MP:
** Merge proposal linked:
https://code.launchpad.net/~dannf/livecd-rootfs/+git/livecd-rootfs/+merge/459549
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to util-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2045586
On Tue, Jan 30, 2024 at 6:21 PM Michael Hudson-Doyle
<2045...@bugs.launchpad.net> wrote:
>
> Oh wait, we've been through something very like this before
> https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1834875. I suspect
> a judicious application of flock may be the most correct solution
>
Oh wait, we've been through something very like this before
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1834875. I suspect
a judicious application of flock may be the most correct solution
available.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded
I updated my livecd-rootfs PPA test package that runs this section of
code in a loop to use this pattern, and it survived until the
autopkgtest timeout - 304 iterations:
https://autopkgtest.ubuntu.com/results/autopkgtest-noble-dannf-
loop/noble/amd64/l/livecd-rootfs/20240128_061640_5c909@/log.gz
I was bothered by the fact that we only sometimes see the double
partition rescan in dmesg. If udev always rescans the partitions of a
full block devices, shouldn't we always see those messages twice?
It turns out that udev doesn't rescan partitions just because a new
block device appears. When
That kernel change doesn't fix the issue:
https://autopkgtest.ubuntu.com/results/autopkgtest-noble-dannf-
loop/noble/amd64/l/livecd-rootfs/20240125_203808_8b5c9@/log.gz
Which actually didn't surprise me after thinking about it. systemd-udevd
is going to ask for a partition reread when it gets
This bug was fixed in the package livecd-rootfs - 24.04.21
---
livecd-rootfs (24.04.21) noble; urgency=medium
* live-build/functions: avoid losetup -P as it appears to race with udev and
do it a bit more by-hand instead. (LP: #2045586)
-- Michael Hudson-Doyle Thu, 25 Jan
Here's my workaround then https://code.launchpad.net/~mwhudson/livecd-
rootfs/+git/livecd-rootfs/+merge/459380
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to util-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2045586
** Merge proposal linked:
https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/459380
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to util-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2045586
On Tue, Jan 23, 2024 at 8:55 PM Michael Hudson-Doyle
<2045...@bugs.launchpad.net> wrote:
>
> Amazing debugging Dann. Until we can get a kernel fix, what's the way
> forward here? Run losetup without -P, run udevadm settle, run partprobe
> on the device (then maybe run udevadm settle again??)
That
Amazing debugging Dann. Until we can get a kernel fix, what's the way
forward here? Run losetup without -P, run udevadm settle, run partprobe
on the device (then maybe run udevadm settle again??)
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages,
** Summary changed:
- livecd-rootfs uses losetup -P for theoretically reliable/synchronous
partition setup but it's not reliable in noble
+ livecd-rootfs uses losetup -P for theoretically reliable/synchronous
partition setup but it's not reliable
--
You received this bug notification because
I ran the above test:
https://autopkgtest.ubuntu.com/results/autopkgtest-jammy-dannf-test/jammy/amd64/l/livecd-rootfs/20240123_035147_6470b@/log.gz
It does appear that systemd-udevd is trying to scan partitions at the
same time as losetup:
1599s ++ losetup --show -f -P -v
I ran into this on jammy/amd64:
https://autopkgtest.ubuntu.com/results/autopkgtest-
jammy/jammy/amd64/l/livecd-rootfs/20240121_173406_e4f9a@/log.gz
I downloaded all of the amd64 failures and searched for this failure
pattern. These were the kernels that were running at the time:
"Linux
> Is only asking kernel to scan the device; to then generate "kernel udev"
> events; for then udev to wakeup and process/emit "udev udev" events; and
> create the required device nodes.
>
It's not udev that creates nodes like /dev/loop1p1 though is it? That's
devtmpfs surely.
--
You received
I don't have a good explanation, but in the past I've "fixed" such races
by adding a `sync "$loop_device"` before using any of the newly created
partitons. Maybe it's worth trying.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed
my expectation is that udev should be running (somewhere, not sure if it
needs to be both the host and the lxd guest) and that it should process
the device using locks https://systemd.io/BLOCK_DEVICE_LOCKING/.
After that is done, the device should be safe to operate on, in a
consistent manner.
Oh. To the question of whether there was a systemd change in this
window: yes absolutely, because this is the point at which the riscv64
builders moved from lgw manually-operated qemu with a 20.04 guest image,
to bos03 openstack-operated qemu with a 22.04 guest image.
Which is also why we've
On Sat, Dec 09, 2023 at 05:13:28PM -, Andy Whitcroft wrote:
> Was there any systemd/udev change in this timeframe? As the device
> files are very much connected to those.
My understanding is that these devices are supposed to be created directly
by the kernel on devtmpfs and NOT via udev,
Was there any systemd/udev change in this timeframe? As the device
files are very much connected to those.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to util-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2045586
Title:
https://launchpad.net/~ubuntu-
cdimage/+livefs/ubuntu/noble/cpc/+build/544490 is a log from a build
with a new livecd-rootfs that spits out more debugging info on failure.
+ sgdisk binary/boot/disk-uefi.ext4 --print
Disk binary/boot/disk-uefi.ext4: 9437184 sectors, 4.5 GiB
Sector size (logical):
Failing build had kernel
Kernel version: Linux bos03-riscv64-014 5.19.0-1021-generic
#23~22.04.1-Ubuntu SMP Thu Jun 22 12:49:35 UTC 2023 riscv64
The build immediately before the first failure had kernel
Kernel version: Linux riscv64-qemu-lgw01-069 5.13.0-1019-generic
#21~20.04.1-Ubuntu SMP Thu
November 16 was 2 days after livecd-rootfs 24.04.4 landed in the noble
release pocket, superseding 24.04.2.
The code delta between 24.04.2 and 24.04.4 includes removal of support
for "legacy" images (SUBPROJECT=legacy) which doesn't apply here; and
some reorganization of code related to
38 matches
Mail list logo
