[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
** Tags added: kernel-da-key -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
I run into the timeout since 2 days. Before that my oneiric experience was quite good. After reading this comments it seems I should be happy that there is a timeout after 61s, but whats causing udev to timeout? Its very annoying to sit and wait for the system - especially when it does boot a lot faster the day before... Any idea how to find the root cause of the timeout? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
Hmm, just ignore my comment... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
Seems related to #625395 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
Just wanted to chime in that I have the exact same problem with udevd killing lvm vgscan. udevd[139]: timeout: killing 'watershed sh -c '/sbin/lvm vgscan; /sbin/lvm vgchange -a y'' What does this imply? Can I just ignore it till you folks fix it ASAP? BTW: It has been a real fun upgrade experience to oneiric: First bitten by bug#858122 on my desktop. Then now, my "KVM server" (the host of several guests) got bitten badly by the same bug (only on the server it hung hard), and then apparently this one. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
To echo davebb, I just downloaded the ubuntu-11.10-server-amd64.iso (md5: f8a0112b7cb5dcd6d564dbe59f18c35f) on 11/23/2011 and had this problem on two different systems. One was a Sun x4100 (AMD 8354, nvidia chipset, no LVM) and one was a Supermicro Intel box, no LVM. In both cases the systems printed out a large number of the 'udev[90]: timeout: killing ' messages and then froze. The console didn't come up for over 45 minutes. I have left it running to see if the host ever comes up. I see the fix is listed for oneiric updates. These errors happened for me on the first boot after an install and the machines don't finish booting. How can I get access to the fix during the install process? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
Sorry if this is the wrong place to ask this, but this bug currently exists on the Ubuntu-11.10-Server (Oneiric) download I just did today 2011-11-17. And cannot seem to even get the server to start. Server is an HP Proliant DL380G5 The image is the iso available for download I have chosen NOT to use an LVM and just do a straight disk partition instead of using the LVM. So I can say it exists even when not using the LVM setup. I tried reading through the notes here to find the fix and says its fixed, but however still is a problem, or do I need to somehow boot off of CD and then make mods to the system. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
Sorry, updates the wrong initramfs. pkill does make a difference, but still doesn't give a booting system. It does get further and i did see some nbx2 firmware errors again, which i didn't see before using pkill. The fix in #17 however works: http://paste.ubuntu.com/679222/ Adding the following lines to init-bottom/udev: . /scripts/functions wait_for_udev And i have a booting system with network. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
Same problem here, ML350 G5 Funny thing is that after the update from 11.04 to 11.10 i had no problems booting, just no network as described in bug 842560. Booted without network about 3-4 times with succes. To try and fixed that i upgraded the firmwares using the hp firmware CD 930. (which seems to work with a 11.10 desktop boot CD). After this i ran into the killing modprobe / read-only fs problem. And i didn't get the system to boot correctly once since then. Since everything was backed up i decided to start with a clean 11.10 install, but ran in to the same problem. Tried to replace the 'udevadm control --exit' in /scripts/init- bottom/udev with 'pkill udevd', but it has no effect. Guess trying to downgrade firmware's is next, but since hp only supplies seperate windows packages that will take a while... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
@John Miller I've reopened https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/842560 which was marked as a duplicate of this udev bug (as I originally mentioned the firmware issue in the description, but it was unrelated), now that this udev problem is sorted the firmware issue can be looked at in more detail. Let's use LP#842560 to track the firmware issue and leave this bug closed out. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
I'm encountering this issue on a Dell PowerEdge 2950 (pretty common server model, we have a ton of them here). Clean install of 11.10 will not boot properly, same with installing 11.04 and then upgrading. It appears to be running the latest udev (173-0ubuntu4). [ 62.944049] bnx2: Can't load firmware file "bnx2/bnx2-mips-06-6.2.1.fw" [ 62.944081] bnx2 :05:00.0: PCI INT A disabled [ 62.944095] bnx2: probe of :05:00.0 failed with error -2 Apologies if I'm missing something, and let me know if I can provide any further info. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
On Sat, Oct 08, 2011 at 08:08:34PM -, Eduard Hasenleithner wrote: > > (I would prefer not to just '|| true' the udevadm call, though, since I > > think having to occasionally reboot the machine again is better than being > > left with extra udev processes running around doing who-knows-what on the > > system.) > Very valid point. But wouldn't it then be better in the case of udevadm- > control failing to do something which gets 100% attention of the user, > and with an explicit diagnosis message describing the problem? Simply > not moving '/dev' to rootfs, and letting the failed boot to reach its > "fate", seems to be a bit too obscure for me. I agree. Could you file a new bug against the initramfs-tools package requesting this? > Being an "expert user", I would also like to have the busybox-shell for > the failure case of course ;) The obvious way to implement this would be to use the panic handler for script failures. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
> (I would prefer not to just '|| true' the udevadm call, though, since I > think having to occasionally reboot the machine again is better than being > left with extra udev processes running around doing who-knows-what on the > system.) Very valid point. But wouldn't it then be better in the case of udevadm- control failing to do something which gets 100% attention of the user, and with an explicit diagnosis message describing the problem? Simply not moving '/dev' to rootfs, and letting the failed boot to reach its "fate", seems to be a bit too obscure for me. Being an "expert user", I would also like to have the busybox-shell for the failure case of course ;) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
On Sat, Oct 08, 2011 at 11:56:50AM -, Eduard Hasenleithner wrote: > Many thanks. My LVM system now starts successfully, albeit still showing > the 60s hang occasionally. Even the supposed-to-be-affected logical > volume is mounted correctly. Just the "/sbin/lvm vgchange -a y" persists > in the process list, but this apparently does not harm. Wow, I didn't expect that to be the case! I still want to keep the other bug report open and pursue the "hung vgchange" problem, anyway; leaving stray processes running in the initramfs after boot is untidy. > One additional note I want to make: I'd set the udevadm-control timeout > at least to 70 seconds. The epoll_wait timeout is reset every time > something happens. So if the "settle time" is bigger than 1 second, the > total timeout would be more than 61 seconds of udevadm-control. Setting > the timeout to 70 seconds, gives us at least 10 seconds "settle time". Ah, that's a good point. Of course, 70 seconds isn't guaranteed to be enough time either; in the pathological case, you need 60*num_workers seconds, because if you get one worker finishing every 59.9 seconds, the udevd timeout will continue to be extended and we'll still have the udevadm control problem. But 61 seconds should be enough for the 99.99% case, and if we start to see reports of problems in the other .01% we can adapt by raising the timeout. (I would prefer not to just '|| true' the udevadm call, though, since I think having to occasionally reboot the machine again is better than being left with extra udev processes running around doing who-knows-what on the system.) > My conclusion is that "udevadm control --exit" (introduced in april) is > still a bit immature, and further races might wait to happen. It's true that there might be further races, but I don't think it's due to immaturity of udevadm control --exit... rather, the races were always there and we were blissfully ignoring them with pkill (and that's where the bug report came from that led us to switch to udevadm in the first place). -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
@Steve, the new udev is working in my vms! thanks. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
Many thanks. My LVM system now starts successfully, albeit still showing the 60s hang occasionally. Even the supposed-to-be-affected logical volume is mounted correctly. Just the "/sbin/lvm vgchange -a y" persists in the process list, but this apparently does not harm. One additional note I want to make: I'd set the udevadm-control timeout at least to 70 seconds. The epoll_wait timeout is reset every time something happens. So if the "settle time" is bigger than 1 second, the total timeout would be more than 61 seconds of udevadm-control. Setting the timeout to 70 seconds, gives us at least 10 seconds "settle time". My conclusion is that "udevadm control --exit" (introduced in april) is still a bit immature, and further races might wait to happen. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
Great work, Steve! So I suppose this doesn't need a release note after all. ** Changed in: udev (Ubuntu Precise) Status: Triaged => Fix Released ** Changed in: ubuntu-release-notes Status: Triaged => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
This bug was fixed in the package udev - 173-0ubuntu4 --- udev (173-0ubuntu4) oneiric; urgency=low * Set a timeout of 61 seconds when calling udevadm control --exit, to ensure that udevd always reaches its *own* timeout first. If udevadm times out before udevd, it exits non-zero, breaking the move of the /dev mount; and if we ignore the udevadm failure then udevd might still be running, which is the original issue that the patch for bug #624469 was trying to address. Raising the timeout doesn't make the boot any slower, it just makes sure we let udevd count the 60 seconds instead of udevadm. LP: #818177. * debian/patches/race-condition-on-exit: Process events before signals in the worker thread, so we don't sit around waiting 60 seconds for an already-dead process. -- Steve LangasekSat, 08 Oct 2011 00:48:54 -0700 ** Changed in: udev (Ubuntu Oneiric) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
** Changed in: udev (Ubuntu Oneiric) Assignee: James Hunt (jamesodhunt) => Steve Langasek (vorlon) ** Changed in: udev (Ubuntu Oneiric) Status: Triaged => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
With the help of some debugging code, I've confirmed that the patch committed to the branch results in a correct (and speedy) exit in the case where a SIGTERM is received before a worker thread has accepted an event for processing. Will push this to the oneiric queue shortly. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
Tried that; doesn't work because the master process already thinks the event is dispatched as soon as it sends it, whether or not the worker process acts on it with recvmsg(). So I guess we need to try to process the event after all. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
The root error is that in worker_new(), at lines 353 ff., the worker threads process signals and udev event handling requests in the same loop. If it happens that both the signal and the event are received close enough together to be returned in the same epoll_wait(), and the udev_monitor handle is listed *first* in the epoll_event list, the worker will receive the event message (which increases the refcount on the udevd master side), then process the signal, and immediately exit without handling the event. This should be handled by processing fd_signal first, and only if SIGTERM was not received, process fd_monitor. This ensures that we always exit ASAP after receiving the signal, instead of going off and processing another event that could take an indeterminate amount of time. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 818177] Re: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread
** Branch linked: lp:~ubuntu-core-dev/ubuntu/oneiric/udev/ubuntu -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/818177 Title: boot failures because 'udevadm exit' times out while udevd waits for an already-dead thread To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-release-notes/+bug/818177/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
