[Bug 1834875] Re: cloud-init growpart race with udev

2020-08-18 Thread Ɓukasz Zemczak
The version of cloud-utils in the proposed pocket of Eoan that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion. ** Tags removed: verification-needed-eoan ** Changed in: cloud-utils (Ubuntu

[Bug 1834875] Re: cloud-init growpart race with udev

2020-07-03 Thread Mathew Hodson
** Changed in: cloud-utils Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage notifications

[Bug 1834875] Re: cloud-init growpart race with udev

2020-06-08 Thread Michael Hudson-Doyle
Anyway given Eoan's imminent demise, maybe we don't care. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage notifications about this bug go

[Bug 1834875] Re: cloud-init growpart race with udev

2020-06-08 Thread Michael Hudson-Doyle
Huh. Is that with cloud-utils 0.31-5-gef42f6b5-0ubuntu1.1 from proposed? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage notifications

[Bug 1834875] Re: cloud-init growpart race with udev

2020-06-01 Thread Pat Viafore
We are seeing some failures still in Eoan (not Focal) with tracebacks involving growpart: 2020-05-29 10:14:29,218 - util.py[DEBUG]: Running module growpart () failed Traceback (most recent call last): File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 851, in _run_modules

[Bug 1834875] Re: cloud-init growpart race with udev

2020-05-12 Thread Mathew Hodson
** Project changed: cloud-init => cloud-initramfs-tools ** Changed in: cloud-initramfs-tools Status: Incomplete => Fix Released ** Bug watch removed: github.com/systemd/systemd/issues #12258 https://github.com/systemd/systemd/issues/12258 -- You received this bug notification because

[Bug 1834875] Re: cloud-init growpart race with udev

2020-05-12 Thread Brian Murray
Hello Tobias, or anyone else affected, Accepted cloud-utils into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud- utils/0.31-5-gef42f6b5-0ubuntu1.1 in a few hours, and then in the -proposed repository. Please help us by testing this new

[Bug 1834875] Re: cloud-init growpart race with udev

2020-05-05 Thread Michael Hudson-Doyle
** Description changed: - On Azure, it happens regularly (20-30%), that cloud-init's growpart - module fails to extend the partition to full size. + [impact] + On Azure, it happens regularly (20-30%), that cloud-init's growpart module fails to extend the partition to full size. Such as in

[Bug 1834875] Re: cloud-init growpart race with udev

2020-04-22 Thread Scott Moser
@Markus, Please open a bug against cloud-initramfs-growroot and reference it here (and reference this bug there). Please also provide a full console log. fwiw, generally speaking cloud-initramfs-growroot has not been necessary since at least 14.04, quite possibly even beefore. -- You received

[Bug 1834875] Re: cloud-init growpart race with udev

2020-04-22 Thread Markus Schade
It seems that the changes to cloud-initramfs-growroot have a side effect that was not present before focal. After initially provisioning an image, the root partition is resized in initramfs, but cannot be mounted afterwards. A second boot to the now resized disk works then without problems.

[Bug 1834875] Re: cloud-init growpart race with udev

2020-04-08 Thread Michael Hudson-Doyle
** No longer affects: linux-azure (Ubuntu) ** No longer affects: systemd (Ubuntu) ** Also affects: cloud-utils (Ubuntu Eoan) Importance: Undecided Status: New ** Also affects: cloud-initramfs-tools (Ubuntu Eoan) Importance: Undecided Status: New -- You received this bug

[Bug 1834875] Re: cloud-init growpart race with udev

2020-04-08 Thread Pat Viafore
Yes, we see it on Eoan and blocking tests, so if you could SRU it there, it'd be much appreciated. We have never seen it on Bionic to my knowledge. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1834875] Re: cloud-init growpart race with udev

2020-04-08 Thread Michael Hudson-Doyle
Great, thanks for confirming. I wonder if we should SRU this. You see it on Eoan I presume? The bug is theoretically present in Bionic too aiui but if it never crops up I don't know if it's worth it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1834875] Re: cloud-init growpart race with udev

2020-04-08 Thread Pat Viafore
Sorry, we were waiting on a bug regarding dictionary keys being modified to resolve before we could verify this. We have not seen this in the past couple of runs in our testing, so I think we're good to proceed. -- You received this bug notification because you are a member of Ubuntu Bugs,

[Bug 1834875] Re: cloud-init growpart race with udev

2020-03-06 Thread Michael Hudson-Doyle
Can someone from CPC confirm that this has fixed the issues in the Azure tests? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-29 Thread Launchpad Bug Tracker
This bug was fixed in the package cloud-utils - 0.31-7-gd99b2d76-0ubuntu1 --- cloud-utils (0.31-7-gd99b2d76-0ubuntu1) focal; urgency=medium * New upstream snapshot. - growpart: add flock support to prevent udev races (LP: #1834875) -- Ryan Harper Tue, 25 Feb 2020 14:41:21

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-25 Thread Launchpad Bug Tracker
This bug was fixed in the package cloud-initramfs-tools - 0.45ubuntu1 --- cloud-initramfs-tools (0.45ubuntu1) focal; urgency=medium * Add dependency on flock for growroot's use of growpart. (LP: #1834875) -- Scott Moser Tue, 25 Feb 2020 13:08:22 -0500 ** Changed in:

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-25 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-utils/+merge/379851 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2020-02-25 Thread Ryan Harper
On Tue, Feb 25, 2020 at 2:35 PM Scott Moser wrote: > this seemed to "just work" for me. > http://paste.ubuntu.com/p/93dWDPZfZT/ Ah, I didn't check that there was an existing ubuntu/devel branch. Sorry. I've pushed a MR here:

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-25 Thread Scott Moser
this seemed to "just work" for me. http://paste.ubuntu.com/p/93dWDPZfZT/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage notifications

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-25 Thread Ryan Harper
@Scott, cloud-utils isn't quite new-upstream-snapshot out of the box; the debian dir does not contain the changelog; however, I think I've got this sorted out. I've a MP I can put up; but it only will show the add of the changelog file. I'll attach a debdiff and a source package. -- You

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-25 Thread Ryan Harper
** Attachment added: "tarball of source package to upload" https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5330895/+files/cloud-utils_0.31-7-gd99b2d76-source.tar.xz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-25 Thread Ryan Harper
** Patch added: "debdiff showing the changes to upload to fix the bug." https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5330894/+files/cloud-utils_0.31-6_to_0.31-7.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-25 Thread Scott Moser
a.) I gave the wrong link. ugh. It should have been: https://code.launchpad.net/~smoser/cloud-initramfs-tools/+git/cloud-initramfs-tools/+merge/379774 b.) the fixed link to 'a' probably makes more sense now. But basically you need a newer cloud-initramfs-tools to adjust for the fact that

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-25 Thread Joshua Powers
@smoser a looks merged b has not had any changes since 2018 and looks up-to-date in Focal. What am I missing? c I see a devel branch, is it assumed that it uses a packaging structure similar to cloud-init or can someone just grab and upload master? -- You received this bug notification

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-24 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~smoser/cloud-initramfs-tools/+git/cloud-initramfs-tools/+merge/379774 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title:

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-24 Thread Scott Moser
The fix is in cloud-utils upstream now. Still to do: a.) review/merge cloud-initramfs-tools pull request https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-utils/+merge/379177 b.) upload cloud-initramfs-tools to focal c.) upload cloud-utils to focal d.) any SRU the order of 'b'

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-24 Thread Scott Moser
** Changed in: cloud-initramfs-tools (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-24 Thread Scott Moser
** Merge proposal linked: https://code.launchpad.net/~raharper/cloud-utils/+git/cloud-utils/+merge/379177 ** Changed in: cloud-utils Status: New => Fix Committed ** Also affects: cloud-utils (Ubuntu) Importance: Undecided Status: New ** Changed in: cloud-utils (Ubuntu)

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-14 Thread Ryan Harper
Here's the upstream changes to growpart I'm suggesting: https://code.launchpad.net/~raharper/cloud-utils/+git/cloud- utils/+merge/379177 I've also proposed on modifications to cloud-init's cc_growpart as a further method to aid debugging if this hit as well as some mitigation around the race.

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-10 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~raharper/ubuntu/+source/cloud-utils/+git/cloud-utils/+merge/378839 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-10 Thread Ryan Harper
Yes, this is my read on the issue as well. The trigger is related to the inotify watch that systemd-udevd puts on the disk. Something that might help that we could try per xnox's comment around use of flock. if growpart were to flock /dev/sda (we need to sort out what flags are needed to

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-09 Thread Michael Hudson-Doyle
> Ah, no I think this might be along right lines: udev is calling blkid > on the _partition_ of course, so it can probe for filesystem etc without > looking at the partition table. After it's done that, it does look for > the partition table so it can read the ID_PART_ENTRY_* values from it, > but

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-06 Thread Pat Viafore
Adding in a passing version of journalctl ** Attachment added: "passing_journalctl_output.txt" https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5326024/+files/passing_journalctl_output.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which is

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-06 Thread Ryan Harper
Cloud-init service starts and will run growpart, etc Feb 06 00:37:26 ubuntu systemd[1]: Starting Initial cloud-init job (pre-networking)... Feb 06 00:37:37 test-xrdpdnvfctsofyygmzan systemd[1]: Starting Initial cloud-init job (metadata service crawler)... Something has modified sdb1

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-06 Thread Pat Viafore
I'm attaching the output of a failing build ** Attachment added: "failing_journalctl_output.txt" https://bugs.launchpad.net/cloud-init/+bug/1834875/+attachment/5325941/+files/failing_journalctl_output.txt -- You received this bug notification because you are a member of Ubuntu Bugs, which

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-03 Thread Michael Hudson-Doyle
On another tangent, I wonder if the change that brought this to light was the switch to booting without initrd. The timing is about right, and fits with the fact that it doesn't occur with the generic kernel (which cannot boot without initrd in Azure). So if someone has an excess of time to test ,

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-02 Thread Pat Viafore
I can get that going, and will report back on this bug when I have more information. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage

[Bug 1834875] Re: cloud-init growpart race with udev

2020-02-02 Thread Michael Hudson-Doyle
Hi, could someone prepare an image with https://paste.ubuntu.com/p/HYGv6m8gCc/ at /etc/systemd/system/systemd- udevd.service.d/udevd-debugging.conf, boot it on azure until it fails and put the journalctl output (probably a few megs) somewhere I can read it? Output from a successful boot would also

[Bug 1834875] Re: cloud-init growpart race with udev

2020-01-30 Thread Michael Hudson-Doyle
Oh yeah and one other thing I don't understand: why udev is processing the partition while sgdisk is still running. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart

[Bug 1834875] Re: cloud-init growpart race with udev

2020-01-30 Thread Michael Hudson-Doyle
Ah, no I think this might be along right lines: udev is calling blkid on the _partition_ of course, so it can probe for filesystem etc without looking at the partition table. After it's done that, it does look for the partition table so it can read the ID_PART_ENTRY_* values from it, but if it

[Bug 1834875] Re: cloud-init growpart race with udev

2020-01-29 Thread Michael Hudson-Doyle
So I've been handed the job of moving this bug forward this iteration. Having just re-read all the comments again and read some source code while listening to loud techno music my observations/questions are these: 1) this is a doozy 2) I would really like to see complete udevadm monitor and

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Dimitri John Ledkov
On Thu, 7 Nov 2019 at 20:05, Scott Moser wrote: > > > > So that means we have this sequence of events: > > > a.) growpart change partition table > > > b.) growpart call partx > > > c.) udev created and events being processed > > > That is not true. whilst sfdisk is deleting, creating,

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Dimitri John Ledkov
On Thu, 7 Nov 2019 at 17:50, Ryan Harper <1834...@bugs.launchpad.net> wrote: > > @ddstreet > > Yes, settle does not help. > > Re-triggering udevadm trigger --action=add /sys/class/block/sda > > Would re-run all of them after the partition change has occurred, which > is what I'm currently

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Dan Streetman
> We can't know how many devices are in the system. It maybe nearly zero, > it could be minutes. I'm not suggesting you trigger uevents for all devices, just the one you're repartitioning... > The cold plug is required (initramfs may or maynot > have ran > scripts/udevd etc but kernel events

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Ryan Harper
On Thu, Nov 7, 2019 at 2:41 PM Dan Streetman wrote: > > Issuing a second > > trigger will repeat this. > > IMO, that's a non-zero amount of time that slows the boot down, so I'd > like > > to avoid that. > > systemd-udev-trigger.serivce retriggers *everything* at boot (except in > an

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Ryan Harper
On Thu, Nov 7, 2019 at 11:30 AM Dimitri John Ledkov wrote: > > So that means we have this sequence of events: > > a.) growpart change partition table > > b.) growpart call partx > > c.) udev created and events being processed > > That is not true. whilst sfdisk is deleting, creating,

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Dan Streetman
> Issuing a second > trigger will repeat this. > IMO, that's a non-zero amount of time that slows the boot down, so I'd like > to avoid that. systemd-udev-trigger.serivce retriggers *everything* at boot (except in an unprivileged container where it can't), so I'm not sure how much added time

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Scott Moser
> > So that means we have this sequence of events: > > a.) growpart change partition table > > b.) growpart call partx > > c.) udev created and events being processed > That is not true. whilst sfdisk is deleting, creating, finishing > partition table (a) and partx is called (b), udev events

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Ryan Harper
On Thu, Nov 7, 2019 at 1:30 PM Dan Streetman wrote: > > Yes, settle does not help. > > Well, I didn't suggest just to settle ;-) > Sorry; long bug thread. > > I'm currently suggesting as a heavy-handed workaround. > > I don't really see why you think this is heavy-handed, but I must be >

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Dan Streetman
> Yes, settle does not help. Well, I didn't suggest just to settle ;-) > I'm currently suggesting as a heavy-handed workaround. I don't really see why you think this is heavy-handed, but I must be missing something. > I would like to understand *why* the udevd/kernel pair exhibits this racy >

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Ryan Harper
@ddstreet Yes, settle does not help. Re-triggering udevadm trigger --action=add /sys/class/block/sda Would re-run all of them after the partition change has occurred, which is what I'm currently suggesting as a heavy-handed workaround. I would like to understand *why* the udevd/kernel pair

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Dan Streetman
Just curious, did anyone actually test with my suggestion from comment 40? That is, just make your partition table change (and update the kernel with partx or whatever, of course), and after it's done trigger --settle udev for the device. Be interesting to know if that actually fixes it or not.

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Dimitri John Ledkov
> So that means we have this sequence of events: > a.) growpart change partition table > b.) growpart call partx > c.) udev created and events being processed That is not true. whilst sfdisk is deleting, creating, finishing partition table (a) and partx is called (b), udev events are already

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Scott Moser
I really think you are all *way* over thinking this. a. growpart made a change to the partition table (using sfdisk) b. growpart called partx --update --nr 3 /dev/sda c. growpart exited With a and b growpart created udev events. If you create udev events, you really need to wait for those

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-07 Thread Ryan Harper
> it will prevent udevd from running the rules against it. Thus effectively the event will be fired and done, but nothing actually executed for it. Interesting, I suspect this is the race we see. The events emitted but no actions taken (ie we didn't get our by-partuuid symlink created. > I

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-06 Thread Dimitri John Ledkov
flock will not block inotify or udev events being emitted. See https://github.com/systemd/systemd/blob/master/src/udev/udevd.c#L322 https://github.com/systemd/systemd/blob/master/src/udev/udevd.c#L409 it will prevent udevd from running the rules against it. Thus effectively the event will be

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-06 Thread Ryan Harper
A couple of comments on the suggested path: > Imho the sequency of commands should be: > * take flock on the device, to neutralise udev +1 on this approach. Do you know if the flock will block systemd's inotify write watch on the block device which triggers udevd? This is the typical race we

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-06 Thread Dimitri John Ledkov
@ Cloud init team, do we want to try changing cloud-utils to use a lock? And like have a canary "only use locked codepath on this region" such that we can assert through testing that this no longer happens with new code, but does with old code. -- You received this bug notification because you

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-01 Thread Dimitri John Ledkov
** Changed in: systemd (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage notifications about this

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-01 Thread Dimitri John Ledkov
Some observations: * growpart uses sfdisk without --no-tell-kernel option, meaning that it does notify kernel about partition changes * growpart later calls partx, which may be redundant / cause no changes or events * as a side note, partprobe, blockdev --rereadpt can also be used to reread

[Bug 1834875] Re: cloud-init growpart race with udev

2019-11-01 Thread Francis Ginther
** Tags added: id-5dbb311b74e3f364d8f98c56 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage notifications about this bug go to:

[Bug 1834875] Re: cloud-init growpart race with udev

2019-09-30 Thread Dan Streetman
> The lack of events is not the issue I'm not saying it is. But you seem to have it under control; I was just offering my opinion. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title:

[Bug 1834875] Re: cloud-init growpart race with udev

2019-09-30 Thread Dan Watkins
The lack of events is not the issue. We see all the events from the kernel that we would expect to see during a resize, and we see the same number (and type) of events on both the passing and the failing instances. The problem is that if udev processes the sda1 event first then, at least some of

[Bug 1834875] Re: cloud-init growpart race with udev

2019-09-30 Thread Dan Streetman
> You're proposing retriggering events and _then_ waiting I'm proposing make sure that the uevent(s) for the specific partition you're interested in have started processing before you start waiting, which is what the retriggering does. But it could be a kernel bug, sure. -- You received this

[Bug 1834875] Re: cloud-init growpart race with udev

2019-09-30 Thread Dan Watkins
I don't think we're looking for a way to wait for udev. For this bug to present, udev has to have processed all of the available events (i.e. there's nothing left to wait for), and have done so incorrectly (probably as a result of its interaction with the Azure kernel, as this doesn't reproduce

[Bug 1834875] Re: cloud-init growpart race with udev

2019-09-30 Thread Dan Streetman
I think that is the correct way to wait for udev to process a specific device's changes. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage

[Bug 1834875] Re: cloud-init growpart race with udev

2019-09-30 Thread Dan Watkins
Dan, are you asking for that to get more debugging information, or are you suggesting that as a fix for this? (I'm yet to be convinced that this isn't a regression in (the kernel|src:systemd), so I don't think that we should have to run _any_ additional commands to address this.) -- You

[Bug 1834875] Re: cloud-init growpart race with udev

2019-09-30 Thread Dan Streetman
Pretty sure you need to actually try what I suggested in comment 40, but of course use the specific dev/partition you're missing the udev symlink for. e.g. if you need the symlink for /dev/sda1 then instead of: $ growpart /dev/sda 1 ; udevadm settle ; ls /dev/disk/by-uuid/$UUID you should do $

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2019-09-03 Thread Dan Watkins
On Mon, Aug 26, 2019 at 03:17:10PM -, Scott Moser wrote: > If there is a race, or a need to wait, it almost certainly is in cloud- > utils (growpart). I would still like us to get a systemd/kernel person to take a look at this, because I think that the race is somewhere further down than

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2019-09-03 Thread Dan Watkins
On Mon, Aug 26, 2019 at 08:58:46AM -, Tobias Koch wrote: > > (Odds are that whatever causes it to be recreated later in boot would be > > blocked by cloud-init waiting.) > > But that's not happening. The instance does boot normally, the only > service degraded is cloud-init and there is no

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-26 Thread Scott Moser
If there is a race, or a need to wait, it almost certainly is in cloud- utils (growpart). If you use growpart to grow a partition, the result should be that that is done, and it is ready. The caller should not expect that they have to know additional information (to call 'udevadm settle') or

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2019-08-26 Thread Ryan Harper
On Mon, Aug 26, 2019 at 4:05 AM Tobias Koch <1834...@bugs.launchpad.net> wrote: > > (Odds are that whatever causes it to be recreated later in boot would be > > blocked by cloud-init waiting.) > > But that's not happening. The instance does boot normally, the only > service degraded is cloud-init

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-26 Thread Scott Moser
** Also affects: cloud-utils Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage notifications

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-26 Thread Tobias Koch
> (Odds are that whatever causes it to be recreated later in boot would be > blocked by cloud-init waiting.) But that's not happening. The instance does boot normally, the only service degraded is cloud-init and there is no significant delay either. So conversely, if I put a loop into cloud-init

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2019-08-23 Thread Dan Watkins
On Fri, Aug 23, 2019 at 08:23:06PM -, Tobias Koch wrote: > I may be missing the point, but the symlink in question is eventually > recreated, does that tell us anything? This here Yes, this is more supporting evidence that this is a race condition; the state of the system both before and some

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-23 Thread Tobias Koch
I may be missing the point, but the symlink in question is eventually recreated, does that tell us anything? This here > Dan had put a udevadm settle in this spot like so > > def get_size(filename) >util.subp(['udevadm', 'settle']) >os.open() looks to me like the event queue should

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2019-08-23 Thread Ryan Harper
On Fri, Aug 23, 2019 at 10:21 AM Dan Watkins wrote: > Looking at the comment timestamps, Dan probably didn't see my comment, > but to reiterate: all the events we expect to be processed _are_ > processed, the issue is that when they are processed they don't always > end up with the correct

Re: [Bug 1834875] Re: cloud-init growpart race with udev

2019-08-23 Thread Ryan Harper
On Fri, Aug 23, 2019 at 10:00 AM Dan Streetman wrote: > > Dan had put a udevadm settle in this spot like so > > > > def get_size(filename) > >util.subp(['udevadm', 'settle']) > >os.open() > > if you know you've just changed (e.g.) /dev/sda, possibly its kernel- > generated udev

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-23 Thread Dan Watkins
Looking at the comment timestamps, Dan probably didn't see my comment, but to reiterate: all the events we expect to be processed _are_ processed, the issue is that when they are processed they don't always end up with the correct partition information. -- You received this bug notification

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-23 Thread Dan Streetman
> Dan had put a udevadm settle in this spot like so > > def get_size(filename) >util.subp(['udevadm', 'settle']) >os.open() if you know you've just changed (e.g.) /dev/sda, possibly its kernel- generated udev events just haven't reached udev yet, so the immediate call to 'udev

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-23 Thread Dan Watkins
[N.B. I wrote the below before I saw Ryan's comment, so there is some repetition.] OK, I've spent some time catching up on this properly so I can summarise: per comment #24, the issue is that when udev processes the events emitted by the kernel, it (sometimes) doesn't determine the correct

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-23 Thread Ryan Harper
The sequence is: exec growpart exec sgdisk --info # read-only exec sgdisk --pretend # read-only exec sgdisk --backup # read-only copy # modification of disk starts exec sgdisk --move-second-header \ --delete=PART \ --new=PART \ --typecode

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-23 Thread Tobias Koch
** Changed in: cloud-init Status: Invalid => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage notifications about this bug go

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-23 Thread Tobias Koch
@daniel-thewatkins, I'm not convinced that this bug is invalid for cloud-init. After reading through all of this again, I still don't understand, what guarantee there is that when `udevadm settle` is called, all relevant events have already been queued. Copying udevadm over, and with that

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-22 Thread Dan Watkins
So, to reset on status: I believe that we've narrowed this down to a systemd/udev problem, as it goes away when a different udevadm binary is used. I don't have a good sense of how to go about digging into that issue, so I don't think I can take it much further. (Also, as it appears to be a

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-07 Thread Tobias Koch
When I say unorthodox, I mean I copied the binary ;) So that should narrow it down quite a bit. Unless there is more funniness involved. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title:

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-06 Thread Dan Watkins
That certainly is unorthodox! When you say "copied udevadm", what specifically do you mean? Just the binary, the udev package, ...? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title:

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-05 Thread Tobias Koch
Oh, good observation, Dan. Yes, it was happening on Cosmic and later. But I cannot say when it started. Now in a moment of despair I tried something very unorthodox: I copied udevadm from Bionic to the Eoan image and ran the tests again. Look and behold, all pass. ** No longer affects:

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-02 Thread Dan Watkins
Now one wrinkle about it being a linux-azure bug: if you _aren't_ seeing this on bionic but _were_ seeing it on cosmic, then there must be something else involved, because I believe (based on the publishing history[0]) that cosmic was on the same kernel version as bionic. This could, of course,

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-02 Thread Dan Watkins
(Oh, and thank you for that test run!) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage notifications about this bug go to:

[Bug 1834875] Re: cloud-init growpart race with udev

2019-08-02 Thread Tobias Koch
** Also affects: linux-azure (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1834875 Title: cloud-init growpart race with udev To manage