** Description changed: + === Begin SRU Template === + [Description] + A race condition can occur when invoking grub-install inside the target environment. + The failure case shows logs like this: + | Installing for i386-pc platform. + | Installation finished. No error reported. + | umount: /tmp/tmpM4R1dI/target/dev: device is busy. + | (In some cases useful info about processes that use + | the device is found by lsof(8) or fuser(1)) + | Unexpected error while running command. + | Command: ['umount', '/tmp/tmpM4R1dI/target/dev'] + | Exit code: 1 + | Reason: - + | Stdout: '' + | Stderr: '' + | Installation failed with exception: Unexpected error while running command. + | Command: ['curtin', 'curthooks'] + + This is believed to be because some process (likely spawned by udev) has + open filehandles on /dev when curtin went to clean up the target mounts. + + The solution is to run 'udevadm settle' before unmounting '/dev/' from + the target. + + [Impact] + The impact is transient failure to install. This race condition is very rarely seen on hardware, but was somewhat easily reproduced in a heavily loaded vmware environment. + + [Test Case] + In the original bug-opener's environment it fails fairly reliably under heavy host load using vmware. He would do a deploy to several guests on the same host at the same time and this would reproduce. Unfortunately I was unable to come up with a test case in a less complex environment. + + [Regression Potential] + Regression potential should be very low here. The most likely fallout is just additional time for the install as a result of running 'udevadm settle'. A system that did not exhibit this bug will install a small fraction of a second slower. + + $ sudo bash -c 'time for x in "$@"; do udevadm settle; done' -- $(seq 1 100) + real 0m0.214s + user 0m0.012s + sys 0m0.008s + + As shown above, that is likely to be on the order of 1/100th of a + second. + + === End SRU Template === + + Here's the relevant part of the curtin output: Installing for i386-pc platform. Installation finished. No error reported. umount: /tmp/tmpM4R1dI/target/dev: device is busy. - (In some cases useful info about processes that use - the device is found by lsof(8) or fuser(1)) + (In some cases useful info about processes that use + the device is found by lsof(8) or fuser(1)) Unexpected error while running command. Command: ['umount', '/tmp/tmpM4R1dI/target/dev'] Exit code: 1 Reason: - Stdout: '' Stderr: '' Installation failed with exception: Unexpected error while running command. Command: ['curtin', 'curthooks'] I spoke with Scott Moser on IRC about this, and he suggested the attached patch, which fixed the problem 100%. I checked the installation output of a few of my MAAS nodes, and didn't see any lsof output. So I assume the "udevadm settle" command is the fix. (Thanks Scott! I hope we can land this in time for MAAS 1.8.)
-- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to curtin in Ubuntu. https://bugs.launchpad.net/bugs/1462139 Title: curtain fails with "Device is busy" error during unmount To manage notifications about this bug go to: https://bugs.launchpad.net/curtin/+bug/1462139/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
