[Bug 525154] Re: mountall for /var races with rpc.statd
Following up on my own question: I don't see any updated mountall or nfs package in natty-proposed, and my fully-updated natty system still is still failing to mount my nfs shares at boot because of a race with statd. When I change this line in /etc/init/mountall-net.conf: start on net-device-up To this: start on net-device-up or stopped statd-mounting mountall gets called again after statd has actually started, and my nfs shares get mounted at startup. I'm attaching a patch for mountall. ** Patch added: mountall-net-nfs-on-statd.patch https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+attachment/2186983/+files/mountall-net-nfs-on-statd.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/525154/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
On 11-07-01 02:52 PM, Forest wrote: Following up on my own question: I don't see any updated mountall or nfs package in natty-proposed, and my fully-updated natty system still is still failing to mount my nfs shares at boot because of a race with statd. My advise here would be either (a) give up on running a /var that's separate from / and just learn to cope with a system that becomes entirely useless at some point because something writing into /var has filled your root filesystem or (b) switch to a different distro that actually pays attention to server deployment practices wherein separating /var from / (and /usr for that matter) is an accepted and supported practice. Given that this bug has existed since Lucid (3 releases now) makes it clear to me that Ubuntu is not at all interested in supporting server deployments where responsible practice is to keep /var from being able to cripple an entire system simply because it fills up. I guess Ubuntu is targeting the desktop and if you want to deploy servers (where you likely will have budgets for support contracts) you need to look at a different distro. Just my perspective having watched this bug stagnate through three releases. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/525154/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I'm still seeing this problem on a fully updated Natty. My nfs mount occasionally succeeds at boot, but most of the time it doesn't. After editing /etc/init/mountall.conf to log the mountall --debug output, I see these messages in the log: mounting /myremotedir spawn: mount -t nfs -o rw,intr,retrans=180,async,noatime,nodiratime 10.0.95.4:/myremotedir /myremotedir mount.nfs: rpc.statd is not running but is required for remote locking. mount.nfs: Either use '-o nolock' to keep locks local, or start statd. mount.nfs: an incorrect mount option was specified mountall: mount /myremotedir [896] terminated with status 32 When the mount fails, status statd reports that statd is running, which makes me think there is still a race condition here. I'm not sure if it's relevant, but the 10.0.95.x network is reachable via eth1, not eth0. I see a lot of fix released notes on this bug report. Has the fix made it in to the Natty release repositories yet? If so, it looks to me like the fix needs some work. ** Also affects: mountall (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/525154/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
On 11-06-20 07:10 PM, Clint Byrum wrote: Christian, if you're using NFS root, you probably have an issue. But it is probably not *this* issue, as this one was not specific to nfs root configurations. It would be quite helpful if you were to raise a new bug report against nfs-utils that detailed what you expect to have happen, and what is actually happening. But the question that begs to be asked is why are common configurations like NFS-root and separate /var and /usr filesystems STILL not part of Ubuntu's standard QA processes? These are not entirely esoteric configurations you know and they have been shown to have problems in past releases so why are current QA processes not testing for these? You do understand that effective QA means that when a configuration is shown to have a potential for regressions that such a configuration be added to the battery of tests that QA runs. It's simply not effective to identify a configuration that doesn't work, (think that you have) fix(ed) it and simply move on and not ever test that configuration again during a regular release cycle. That's exactly how regressions leak into GA product. It's embarrassing. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
Excerpts from Brian J. Murrell's message of Tue Jun 21 10:18:06 UTC 2011: On 11-06-20 07:10 PM, Clint Byrum wrote: Christian, if you're using NFS root, you probably have an issue. But it is probably not *this* issue, as this one was not specific to nfs root configurations. It would be quite helpful if you were to raise a new bug report against nfs-utils that detailed what you expect to have happen, and what is actually happening. But the question that begs to be asked is why are common configurations like NFS-root and separate /var and /usr filesystems STILL not part of Ubuntu's standard QA processes? These are not entirely esoteric configurations you know and they have been shown to have problems in past releases so why are current QA processes not testing for these? You do understand that effective QA means that when a configuration is shown to have a potential for regressions that such a configuration be added to the battery of tests that QA runs. It's simply not effective to identify a configuration that doesn't work, (think that you have) fix(ed) it and simply move on and not ever test that configuration again during a regular release cycle. That's exactly how regressions leak into GA product. It's embarrassing. Brian, much of our QA is still community driven. We devote significant resources to testing the base system, but multi-server setups like NFS root are taxing to manually test, and more complex to automate. I'd love to say we write a regression test for every issue we fix and run it on every possible configuration. Clearly, we don't. If NFS root is important to you, I would suggest that you help us out by gathering other interested users, and putting together a blueprint for the next UDS. Lets get automated tests setup for this configuration. I'd support this 100%, but I don't think we can do it without some help from the actual users of NFS root systems. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I can definitely file a new bug. I've been on and off fighting this and found that while the solution posted to this bug does not fix this problem with our diskless root, but a related fix does. Here's my statd- starting script below; I noticed that the script is run multiple times and that the start statd line isn't actually syncronous, problems which the rpcinfo check solves. I'm not sure you can assume portmap is listening on localhost, but this works for me: description^ITrigger a statd run start on mounting TYPE=nfs task console output script # This apparently is necessary to ensure the statd run completes; # it's a hack but it seems to work more reliably than anything else while ! rpcinfo -u localhost status; do start statd echo Waiting for statd to show up.. sleep 1s done end script -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
By using the script above, I don't actually need any start on clause in my statd.conf file at all. I posted some stream-of-consciousness entries here: - http://www.async.com.br/~kiko/diary.html?date=20.06.2011 - http://www.async.com.br/~kiko/diary.html?date=21.06.2011 - http://www.async.com.br/~kiko/diary.html?date=22.06.2011 I've pushed the bzr branch containing my complete init setup to http://bazaar.launchpad.net/~kiko/+junk/init-diskless/files -- feel free to poach and comment. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Clint, similarly to Janusz I see these exact same issues on my Natty NFS-root system, even when running with the pristing statd* *portmap* init configuration files. I don't quite understand enough about upstart to say what's wrong yet. I just spent an afternoon chasing this down and am pretty sure a bug still remains somewhere, though I'm finding it hard to see where without complete event logging or an infinite kernel scrollback buffer. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Christian, if you're using NFS root, you probably have an issue. But it is probably not *this* issue, as this one was not specific to nfs root configurations. It would be quite helpful if you were to raise a new bug report against nfs-utils that detailed what you expect to have happen, and what is actually happening. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
well restarting portmap and nfs helped, and i don't want now to revert back to faulty config. i was getting this mesage when booting my diskless workstations: mount.nfs: rpc.statd is not running but is required for remote locking. Either use '-o nolocks' to keep locks local, or start statd. /root was mounted OK of course, because if not, it wouldn't boot at all - problem was with /home and other directories mounted by mountall init scripts - after complete boot, in gdm, there was no /home , i had to once again invoke mount -a (when gdm was running) - i solved it by using rc.local dirty solution ;) portmap and nfs services were starting, but this error message was showing up anyway thanks to this thread i found out, that maybe starting statd was racing with mounting /var by nfs in read write mode or something like this, so i thought that restarting portmap and nfs services in single user mode, before any extra NFS shares are mounted (/home) will solve the problem, and it works fine now. no error messages during boot process. i don't really know if it qualifies for new bug or not. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
Excerpts from Janusz Mordarski's message of Sat Apr 09 16:46:18 UTC 2011: well restarting portmap and nfs helped, and i don't want now to revert back to faulty config. i was getting this mesage when booting my diskless workstations: mount.nfs: rpc.statd is not running but is required for remote locking. Either use '-o nolocks' to keep locks local, or start statd. /root was mounted OK of course, because if not, it wouldn't boot at all - problem was with /home and other directories mounted by mountall init scripts - after complete boot, in gdm, there was no /home , i had to once again invoke mount -a (when gdm was running) - i solved it by using rc.local dirty solution ;) portmap and nfs services were starting, but this error message was showing up anyway thanks to this thread i found out, that maybe starting statd was racing with mounting /var by nfs in read write mode or something like this, so i thought that restarting portmap and nfs services in single user mode, before any extra NFS shares are mounted (/home) will solve the problem, and it works fine now. no error messages during boot process. i don't really know if it qualifies for new bug or not. Janusz, this bug is about statd not being available when NFS mounts are made. This should be fixed in Lucid and Maverick. Make sure all updates are applied. If you have customized any of the upstart jobs in /etc/init that control statd or portmap, that may be causing problems. Look for files with the extension '.dpkg-new', like /etc/init/statd.conf.dpkg-new, if those exist, you will want to make sure to merge them into the live files (like /etc/init/statd.conf). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
Excerpts from Janusz Mordarski's message of Tue Mar 29 09:55:14 UTC 2011: i'm using Ubuntu 10.04 as nfsroot on diskless workstation. For now, only working patch for this issue is putting restart portmap restart nfs in some script started early in /etc/rcS.d (so 'single user' mode stage of init) Janus, can you explain how you think this is related to the bug you've commented on? It sounds like you have a different situation, and you should open a new report or look through some of the other ones against mountall. There are known issues with nfs root that we haven't addressed yet, though we'd like to and it will help if we can have enough information from users like yourself. If you do open another report, it would be a good idea to come back and note the new bug # in the comments here. -- You received this bug notification because you are a direct subscriber of a duplicate bug (692793). https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd Status in “nfs-utils” package in Ubuntu: Fix Released Status in “portmap” package in Ubuntu: Fix Released Status in “nfs-utils” source package in Lucid: Fix Released Status in “portmap” source package in Lucid: Fix Released Status in “nfs-utils” source package in Maverick: Fix Released Status in “portmap” source package in Maverick: Fix Released Status in “nfs-utils” source package in Natty: Fix Released Status in “portmap” source package in Natty: Fix Released Bug description: If one has /var (or /var/lib or /var/lib/nfs for that matter) on its own filesystem the statd.conf start races with the mounting of /var as rpc.statd needs /var/lib/nfs to be available in order to work. I am sure this is not the only occurrence of this type of problem. A knee-jerk solution is to simply spin in statd.conf waiting for /var/lib/nfs to be available, but polling sucks, especially for something like upstart whose whole purpose is to be an event driven action manager. SRU justification: NFS mounts do not start reliably on boot in lucid and maverick (depending on the filesystem layout of the client system) due to race conditions in the startup of statd. This should be fixed so users of the latest LTS can make reliable use of NFS. Regression potential: Some systems may fail to mount NFS filesystems at boot time that didn't fail before. Some systems may hang at boot. Some systems may hang while upgrading the packages (this version or in a future SRU). I believe the natty update adequately guards against all of these possibilities, but the risk is there. TEST CASE: 1. Configure a system with /var as a separate partition. 2. Add one or more mounts of type 'nfs' to /etc/fstab. 3. Boot the system. 4. Verify whether statd has started (status statd) and whether all NFS filesystems have been mounted. 5. Repeat 3-4 until the race condition is triggered. 6. Upgrade to the new version of portmap and nfs-common from -proposed. 7. Repeat steps 3-4 until satisfied that statd now starts reliably and all non-gss-authenticated NFSv3 filesystems mount correctly at boot time. To unsubscribe from this bug, go to: https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+subscribe -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
i'm using Ubuntu 10.04 as nfsroot on diskless workstation. For now, only working patch for this issue is putting restart portmap restart nfs in some script started early in /etc/rcS.d (so 'single user' mode stage of init) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
@All: Did anyone tried to install now nfs-common (in lucid with the latest sru bugfix) in a chroot environment? If so, did nobody see nfs-common failing in nfs-common.postinst? Normally, in a chroot, starting services is disabled and/or not allowed (or in some cases the startup mechanism is somehow diverted to /bin/true or /bin/false whatever). But now, nfs-common.postinst does an invoke.rc statd and this failes in a chroot environment which means, that the package in question is not configured properly. The upstart dependency (started portmap ON_BOOT= or (local-filesystems and started portmap ON_BOOT=y)) doesn't apply inside a chroot. One thing that needs to be done is, to not start the services during installation (which means removing all blind magic of dh_installinit) or whatever it takes to come back with the old behaviour. Regards, \sh -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
On Mon, 2011-02-21 at 14:10 +, Stephan Adig wrote: @All: Did anyone tried to install now nfs-common (in lucid with the latest sru bugfix) in a chroot environment? If so, did nobody see nfs-common failing in nfs-common.postinst? Normally, in a chroot, starting services is disabled and/or not allowed (or in some cases the startup mechanism is somehow diverted to /bin/true or /bin/false whatever). But now, nfs-common.postinst does an invoke.rc statd and this failes in a chroot environment which means, that the package in question is not configured properly. The upstart dependency (started portmap ON_BOOT= or (local-filesystems and started portmap ON_BOOT=y)) doesn't apply inside a chroot. One thing that needs to be done is, to not start the services during installation (which means removing all blind magic of dh_installinit) or whatever it takes to come back with the old behaviour. Hi Stephan, thanks for the feedback. First off, services are always started/stopped with invoke-rc.d on upgrade in Debian packages and, thusly, have always been started and stopped in Ubuntu. This is a known issue in upstart which is under active development. Basically upstart doesn't know anything about the chroot environment, so it doesn't read the /etc/init from the chroot. Take a look at bug #430224 for more information. A simple workaround to get the postinst to succeed is to edit the postinst in /var/lib/dpkg/info/nfs-common.postinst and remove the calls to invoke-rc.d -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
On Mon, Feb 21, 2011 at 02:10:11PM -, Stephan Adig wrote: Did anyone tried to install now nfs-common (in lucid with the latest sru bugfix) in a chroot environment? If so, did nobody see nfs-common failing in nfs-common.postinst? Normally, in a chroot, starting services is disabled and/or not allowed (or in some cases the startup mechanism is somehow diverted to /bin/true or /bin/false whatever). But now, nfs-common.postinst does an invoke.rc statd and this failes in a chroot environment which means, that the package in question is not configured properly. The upstart dependency (started portmap ON_BOOT= or (local-filesystems and started portmap ON_BOOT=y)) doesn't apply inside a chroot. invoke-rc.d should not fail in a chroot environment. How have you configured your chroot? *Any* of the standard methods for disabling services in a chroot should have the intended effect (i.e., diverting /sbin/initctl to /bin/true, or configuring a policy-rc.d to disallow service starting). If you have /sbin/initctl diverted to /bin/*false*, that is a misconfiguration in your environment. One thing that needs to be done is, to not start the services during installation (which means removing all blind magic of dh_installinit) or whatever it takes to come back with the old behaviour. No, that is not a thing that needs to be done. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
You may also refer to LP #643289 posts #2, #3, #4, which provide fixes for mountall blocking and for various start-up problems of statd as well as for idmapd/gssd/rpc_pipefs. Those fixes are based on the above fixes, but go further to make sure mountall and rpc_pipefs mounting proceed in proper sequence where needed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
This bug was fixed in the package portmap - 6.0.0-2ubuntu1.1 --- portmap (6.0.0-2ubuntu1.1) maverick-proposed; urgency=low * debian/upstart renamed to debian/portmap.portmap.upstart, debian/portmap.portmap-boot.upstart, debian/rules: Added to set special ON_BOOT flag during boot, which allows statd to use an AND with 'started portmap ON_BOOT=y'. This version of portmap is a dependency of nfs-utils to fix LP: #525154 * debian/portmap.portmap-wait.upstart: job to wait for portmap to finish starting. also depended on on by nfs-utils. -- Steve Langasek steve.langa...@ubuntu.com Tue, 18 Jan 2011 15:28:05 -0800 ** Changed in: portmap (Ubuntu Maverick) Status: Fix Committed = Fix Released ** Changed in: nfs-utils (Ubuntu Maverick) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
This bug was fixed in the package nfs-utils - 1:1.2.2-1ubuntu1.1 --- nfs-utils (1:1.2.2-1ubuntu1.1) maverick-proposed; urgency=low * debian/nfs-common.statd.upstart, debian/nfs-common.statd-mounting.upstart: refactor startup to wait for local-filesystems. (LP: #525154) * debian/control: depend on portmap version that sets ON_BOOT=y and has the portmap-wait job. * debian/rules: install new statd-mounting upstart job * debian/nfs-common.rpc_pipefs.upstart: instantiate this job separately for gssd and idmapd, so that the filesystem gets mounted and unmounted correctly even if both of gssd and idmapd aren't being run, or if one of the two tries to start before the filesystem is fully mounted. Though it may be simpler now to move this logic back into the gssd and idmapd jobs directly, leave that for a later date. -- Steve Langasek steve.langa...@ubuntu.com Wed, 19 Jan 2011 16:05:07 -0800 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
This bug was fixed in the package nfs-utils - 1:1.2.0-4ubuntu4.1 --- nfs-utils (1:1.2.0-4ubuntu4.1) lucid-proposed; urgency=low * debian/nfs-common.statd.upstart, debian/nfs-common.statd-mounting.upstart: refactor startup to wait for local-filesystems. (LP: #525154) * debian/control: depend on portmap version that sets ON_BOOT=y and has the portmap-wait job. * debian/rules: install new statd-mounting upstart job * debian/nfs-common.rpc_pipefs.upstart: instantiate this job separately for gssd and idmapd, so that the filesystem gets mounted and unmounted correctly even if both of gssd and idmapd aren't being run, or if one of the two tries to start before the filesystem is fully mounted. Though it may be simpler now to move this logic back into the gssd and idmapd jobs directly, leave that for a later date. -- Steve Langasek steve.langa...@ubuntu.com Wed, 19 Jan 2011 16:28:35 -0800 ** Changed in: nfs-utils (Ubuntu Lucid) Status: Fix Committed = Fix Released ** Changed in: portmap (Ubuntu Lucid) Status: Fix Committed = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
This bug was fixed in the package portmap - 6.0.0-1ubuntu2.1 --- portmap (6.0.0-1ubuntu2.1) lucid-proposed; urgency=low * debian/upstart renamed to debian/portmap.portmap.upstart, debian/portmap.portmap-boot.upstart, debian/rules: Added to set special ON_BOOT flag during boot, which allows statd to use an AND with 'started portmap ON_BOOT=y'. This version of portmap is a dependency of nfs-utils to fix LP: #525154 * debian/portmap.portmap-wait.upstart: job to wait for portmap to finish starting. also depended on on by nfs-utils. -- Steve Langasek steve.langa...@ubuntu.com Tue, 18 Jan 2011 15:44:43 -0800 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Accepted portmap into maverick-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance! ** Changed in: portmap (Ubuntu Maverick) Status: Triaged = Fix Committed ** Tags removed: verification-done ** Tags added: verification-needed ** Tags added: verification-done ** Changed in: nfs-utils (Ubuntu Maverick) Status: Triaged = Fix Committed ** Tags removed: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Branch linked: lp:ubuntu/maverick-proposed/portmap -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Branch linked: lp:ubuntu/maverick-proposed/nfs-utils -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Changed in: nfs-utils (Ubuntu Natty) Assignee: Clint Byrum (clint-fewbar) = (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I confirm that after installing portmap and nfs-common packages from maverick-proposed I can use NFS auto-mounting with no problems at all. My /var/log/boot.log has no statd error messages now after testing with two reboots. Thanks! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Tags removed: verification-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
Ditto. The maverick-proposed packages seem to work for me too. After a reboot rpc.statd is indeed started. Cheers! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
After Etienne and Clint said this problem is most easily reproduced by a slow fsck - I checked and I do appear to have filesystems unmounted uncleanly. So I'll investigate that as a separate issue (when I find out how to get my shutdown messages recorded to log). I'm certainly experiencing shutdowns that are very quick compared to my last install in Hardy Heron. /var/log/boot.log: fsck from util-linux-ng 2.17.2 ...etc... init: statd main process (920) terminated with status 1 init: statd main process ended, respawning init: statd main process (930) terminated with status 1 ...etc... /dev/sda1: recovering journal init: statd main process (987) terminated with status 1 init: statd main process ended, respawning /dev/mapper/ubuntu01vg-homelv: recovering journal init: statd main process (993) terminated with status 1 init: statd respawning too fast, stopped /dev/mapper/ubuntu01vg-usermedialv: clean, 12/610800 files, 76473/2441216 blocks /dev/mapper/ubuntu01vg-usrlv: recovering journal /dev/mapper/ubuntu01vg-varlv: recovering journal /dev/mapper/ubuntu01vg-rootlv: clean, 13556/183264 files, 148953/732160 blocks /dev/mapper/ubuntu01vg-tmplv: recovering journal /dev/mapper/ubuntu01vg-usrlocallv: clean, 43/60928 files, 8262/243712 blocks (check in 3 mounts) init: ureadahead-other main process (1040) terminated with status 4 /dev/mapper/ubuntu01vg-tmplv: clean, 785/121920 files, 35321/487424 blocks init: ureadahead-other main process (1061) terminated with status 4 init: mounted-tmp main process (1062) terminated with status 127 mountall: Event failed ...etc... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Verified here with the packages in Lucid proposed. Not only does statd successfully start, but it does so the first time it tries, avoiding bug #697473. Thanks for the fix! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Tags added: verification-done ** Tags removed: verification-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Etienne, thanks for trying it out. I wasn't able to reliably cause statd to lose the race w/ mountall without forcing FS corruption or introducing a sleep into the fsck process. I did so by moving /sbin/fsck to /sbin/fsck.real and then using this script: #!/bin/sh echo sleeping 5 seconds before calling real fsck sleep 5 exec /sbin/fsck.real $@ Given that this is a race condition brought on by fsck problems, the only real world examples are exactly what you had to use, caused by recovering from some other problem. I don't think your method is synthetic at all, but it would be good to get somebody else to verify in a similar manner. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
My machine *always* starts with statd down and therefore with all NFS auto-mounting not working. $ sudo status statd statd stop/waiting I'll look out for maverick-proposed packages to test. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
On Fri, 2011-01-21 at 20:43 +, Ray Nichols wrote: My machine *always* starts with statd down and therefore with all NFS auto-mounting not working. Indeed. Mine too. My lucid machines also. I'll look out for maverick-proposed packages to test. Indeed, me too. I'd rather not diddle with my lucid machines. There is a reason they are LTS installs. :-) Once I see happiness on maverick-proposed I will have more confidence in diddling the lucid machines. b. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Accepted nfs-utils into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance! ** Changed in: nfs-utils (Ubuntu Lucid) Status: Triaged = Fix Committed ** Tags added: verification-needed ** Changed in: portmap (Ubuntu Lucid) Status: Triaged = Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Branch linked: lp:ubuntu/lucid-proposed/portmap -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Branch linked: lp:ubuntu/lucid-proposed/nfs-utils -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I am testing on a VM, and the race is kinda hard to trigger. Mounting of /var needs to be delayed somehow, for example, by an fsck. In my case, bug #579858 trigger an fsck on reboot, which make the race happen somewhat reliably. After updating the nfs-common and portmap packages to the one in lucid- proposed, statd start reliably. So, for my synthetic test-case, it works perfectly. I am not going to set the bug verification-done just yet, as I think it would be best if it was confirmed by someone with a real-world test-case. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Description changed: - Binary package hint: upstart - If one has /var (or /var/lib or /var/lib/nfs for that matter) on its own filesystem the statd.conf start races with the mounting of /var as rpc.statd needs /var/lib/nfs to be available in order to work. I am sure this is not the only occurrence of this type of problem. A knee-jerk solution is to simply spin in statd.conf waiting for /var/lib/nfs to be available, but polling sucks, especially for something like upstart whose whole purpose is to be an event driven action manager. + + SRU justification: NFS mounts do not start reliably on boot in lucid and + maverick (depending on the filesystem layout of the client system) due + to race conditions in the startup of statd. This should be fixed so + users of the latest LTS can make reliable use of NFS. + + Regression potential: Some systems may fail to mount NFS filesystems at + boot time that didn't fail before. Some systems may hang at boot. Some + systems may hang while upgrading the packages (this version or in a + future SRU). I believe the natty update adequately guards against all + of these possibilities, but the risk is there. + + TEST CASE: + 1. Configure a system with /var as a separate partition. + 2. Add one or more mounts of type 'nfs' to /etc/fstab. + 3. Boot the system. + 4. Verify whether statd has started (status statd) and whether all NFS filesystems have been mounted. + 5. Repeat 3-4 until the race condition is triggered. + 6. Upgrade to the new version of portmap and nfs-common from -proposed. + 7. Repeat steps 3-4 until satisfied that statd now starts reliably and all non-gss-authenticated NFSv3 filesystems mount correctly at boot time. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Fixed this by making upstart watch ower the rpc.idmapd. Might be bit cleaner. ** Patch added: idmapd.conf.diff https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+attachment/1796074/+files/idmapd.conf.diff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
You seem to be following up to the wrong bug. This bug is about statd, not idmapd. In general we want to avoid running services in the foreground. The idmapd bug is probably fixable without resorting to such a change. We should probably fix idmapd to not fork before it's propery running, at any rate. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Changed in: nfs-utils (Ubuntu Lucid) Status: Confirmed = Triaged ** Changed in: portmap (Ubuntu Lucid) Status: New = Triaged ** Changed in: nfs-utils (Ubuntu Maverick) Importance: Undecided = High ** Changed in: nfs-utils (Ubuntu Lucid) Importance: Undecided = High ** Changed in: portmap (Ubuntu Lucid) Importance: Undecided = High ** Changed in: portmap (Ubuntu Maverick) Importance: Undecided = High ** Changed in: portmap (Ubuntu Maverick) Status: New = Triaged ** Changed in: nfs-utils (Ubuntu Maverick) Status: Confirmed = Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
This bug was fixed in the package portmap - 6.0.0-2ubuntu2 --- portmap (6.0.0-2ubuntu2) natty; urgency=low * debian/upstart renamed to debian/portmap.portmap.upstart, debian/portmap.portmap-boot.upstart, debian/rules: Added to set special ON_BOOT flag during boot, which allows statd to use an AND with 'started portmap ON_BOOT=y'. This version of portmap is a dependency of nfs-utils to fix LP: #525154 * debian/portmap.portmap-wait.upstart: job to wait for portmap to finish starting. also dependedon on by nfs-utils. -- Clint Byrum cl...@ubuntu.com Wed, 05 Jan 2011 11:47:26 -0800 ** Changed in: portmap (Ubuntu Natty) Status: Confirmed = Fix Released ** Changed in: nfs-utils (Ubuntu Natty) Status: In Progress = Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
This bug was fixed in the package nfs-utils - 1:1.2.2-4ubuntu2 --- nfs-utils (1:1.2.2-4ubuntu2) natty; urgency=low * debian/nfs-common.statd.upstart, debian/nfs-common.statd-mounting.upstart: refactor startup to wait for local-filesystems. (LP: #525154) * debian/control: depend on portmap version that sets ON_BOOT=y and has the portmap-wait job. * debian/rules: install new statd-mounting upstart job -- Clint Byrum cl...@ubuntu.com Wed, 05 Jan 2011 12:27:32 -0800 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Changed in: nfs-utils (Ubuntu Natty) Status: Triaged = In Progress ** Changed in: nfs-utils (Ubuntu Natty) Assignee: (unassigned) = Clint Byrum (clint-fewbar) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Branch linked: lp:~clint-fewbar/ubuntu/natty/nfs-utils/wait-for- local-filesystems -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Also affects: portmap (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Branch linked: lp:~clint-fewbar/ubuntu/natty/portmap/new-boot-events -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Alright I've pushed up branches for nfs-utils and portmap that address this issue in natty. The open merge proposal for nfs-utils is dependent on the portmap package due to the new portmap-wait and portmap-boot upstart jobs. I've tested this on a clean natty vm with and without sleeps inserted for fsck, portmap start, and statd start, and it all seems to work no matter the race. Note that I was able to get around looping and polling for upstart jobs to start with the technique used in portmap-wait and statd-mounting. This seems to be a reliable way to allow waiting on a job that one did not trigger the start/stop for, and may be a good candidate for an upstart feature request. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Changed in: nfs-utils (Ubuntu Lucid) Status: New = Confirmed ** Changed in: nfs-utils (Ubuntu Maverick) Status: New = Confirmed ** Changed in: portmap (Ubuntu Natty) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
On Wed, 2011-01-05 at 20:34 +, Clint Byrum wrote: Alright I've pushed up branches for nfs-utils and portmap that address this issue in natty. The open merge proposal for nfs-utils is dependent on the portmap package due to the new portmap-wait and portmap-boot upstart jobs. I've yet to look at the branch to see what changes have been made but perhaps you can tell me, is this accomplished all with modifications to upstart scripts or other feasibly editable (i.e. text) files? If so, I can try porting your changes back to one of my several maverick (or lucid even -- this should get backported to lucid, which is the LTS release, afterall ---in any case, right?) systems that has problems with statd not getting started properly. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
Yes this is all done in upstart mods. They are fairly extensive.. And can potentially make the system unbootable if done wrong, so do be careful. The fact that this is already accepted for maverick and lucid means that the fix will be backported. On Jan 5, 2011, at 2:56 PM, Brian J. Murrell br...@interlinx.bc.ca wrote: On Wed, 2011-01-05 at 20:34 +, Clint Byrum wrote: Alright I've pushed up branches for nfs-utils and portmap that address this issue in natty. The open merge proposal for nfs-utils is dependent on the portmap package due to the new portmap-wait and portmap-boot upstart jobs. I've yet to look at the branch to see what changes have been made but perhaps you can tell me, is this accomplished all with modifications to upstart scripts or other feasibly editable (i.e. text) files? If so, I can try porting your changes back to one of my several maverick (or lucid even -- this should get backported to lucid, which is the LTS release, afterall ---in any case, right?) systems that has problems with statd not getting started properly. -- You received this bug notification because you are a bug assignee. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd Status in “nfs-utils” package in Ubuntu: In Progress Status in “portmap” package in Ubuntu: Confirmed Status in “nfs-utils” source package in Lucid: Confirmed Status in “portmap” source package in Lucid: New Status in “nfs-utils” source package in Maverick: Confirmed Status in “portmap” source package in Maverick: New Status in “nfs-utils” source package in Natty: In Progress Status in “portmap” source package in Natty: Confirmed Bug description: Binary package hint: upstart If one has /var (or /var/lib or /var/lib/nfs for that matter) on its own filesystem the statd.conf start races with the mounting of /var as rpc.statd needs /var/lib/nfs to be available in order to work. I am sure this is not the only occurrence of this type of problem. A knee-jerk solution is to simply spin in statd.conf waiting for /var/lib/nfs to be available, but polling sucks, especially for something like upstart whose whole purpose is to be an event driven action manager. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Changed in: nfs-utils (Ubuntu) Importance: Undecided = High ** Also affects: nfs-utils (Ubuntu Lucid) Importance: Undecided Status: New ** Also affects: nfs-utils (Ubuntu Maverick) Importance: Undecided Status: New ** Also affects: nfs-utils (Ubuntu Natty) Importance: High Status: Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I must back #62. That is the same situation I am in. The problem that sometimes NFS mounts work an sometimes not is not acceptable. And that is not that our setup is that problematic. Just a /home from NFS. The behavior of upstart is far from reliability. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/525154 Title: mountall for /var races with rpc.statd -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
The attached patch changes the rpc_pipefs upstart job to start on statd and also poll in the pre-start script for the directory /var/lib/nfs/rpc_pipefs for 120*0.5 seconds to become available. Fixes my problem booting a NFSv4 Lucid server and survives an apt-get install --reinstall nfs-common nfs-kernel-server of the patched nfs- utils packages. Hope that the /var fsck / mount never takes more then 60 seconds (didn't check that yet). ** Patch added: Fixed rpc_pipefs upstart job. https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+attachment/1751215/+files/rpc_pipefs.conf.diff -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
For those for whom workarounds in comments #8 and #13 do not work and who use autofs, there is a separate Upstart-induced problem related to statd: it races not only with mountall, but also with autofs. Bug 573919 has more information. Autofs users should also see bug 579858 if they wish to have their local drives cleanly unmounted on shutdown. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
As written in #46 we have also this intolerable boot problem. (Like others) my two colleagues and I have insisted on building our systems on Ubuntu Server instead of SUSE Professional which redounds upon us. In the meantime I have worked on this problem for many weeks. So time was going to run out and I was going to lose my temper. This week (as a last try before switching to another distribution) I made again a new instalIation (after countless other tries). I used no separate /var-partition and tried to change the NFS-entries in the /etc/fstab to noauto. In consequence of this I couldn't use “mount –a –t nfs” for mounting these directories. As a first primitive approach I mounted those in /etc/rc.local by calling each mount command directly after a sleep period. This solution worked, but was not useable in practice. So I made a script which is scanning the /etc/fstab for NFS-entries and is mounting / unmounting (here in reverse order) the directories. Before mounting after boot there is a 20 seconds sleep period to be sure all necessary initializations have be done. The script (placed in /etc/init.d) is invoked over links from /etc/rc?.d. It works well. If this solution is compatible with a separate /var-partition I haven’t tested yet … -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I also have a separate /var filesystem and had a problem with autofs, which was cured by the solution proposed in #13. Is there any news about this bug? -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Hmmm, that suggests there are two or three problems here: 1) upstart cannot correctly identify when it is safe to start certain jobs 2) rpc.statd may not fail cleanly if launched in an unsafe system state (maybe) 3) mountall doesn't always respond to SIGUSR1 (maybe) I'm not observing (2) -- but I'm not running a real server, so my observations may be simplistic. Are you in a position to debug what's going on with statd? For (3), it's a long shot, but this might be related to an old known issue affecting responsiveness to signals which I observed in the libnih code, but never observed at runtime - see https://bugs.launchpad.net/ubuntu/+source/libnih/+bug/518921. AFAICT from the code, this issue still potentially exists. @David: It would be interesting if you could try my fix for this to see if it makes any difference for you. (You can download updated libnih and mountall packages from my PPA https://launchpad.net/~dave-martin- arm/+archive/ppa, assuming they've finished building.) Using start on filesystem and ... seems to give me a working statd in that clients don't hang, but I guess since you have NFS mounts in fstab this won't work in your case. With the default start condition, I get no statd running at all, but can manually do start statd and that works too. I don't seem to end up with the half-working state you saw in this configuration. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
particularly @Scott, @Steve
Since I've now hit this bug again, I had another read over the bug
thread...
Here are some thoughts... which may be substantially wrong, but hey.
There feels like a disconnect here between the true preconditions for some
jobs, and the kind of preconditions specifyable for upstart jobs in general.
A fundamental problem, if I understand the situation correctly, is that
we have cases where the events (things happening dynamically during
boot) are not adequate to determine whether/when upstart should consider
a job startable, at least at the level of the simple boolean
combinations etc. that upstart currently understands.
The startability of some jobs depends on other factors (in this case,
static system configuration which the administrator expects to customise
--- the fstab). If upstart is conservative and waits until _everything_
is mounted, we will fail in some cases, for example when there are NFS
mounts in fstab. Alternatively, if upstart is aggressive and tries to
start the statd job as soon as it is _probably_ startable, then it might
fail to start, and there's not much we can do about it -- that seems to
be the current behaviour.
This is a problem because upstart doesn't currently have any sensible
metholodogy for retrying failed jobs. So, we either need a way to retry
jobs at sensible times, or we would need a more expressive way to
determine when jobs should be started.
Conservative approach
==
The conservative approach would be this approximation (which seems work for
me):
start on (filesystem and (started portmap or mounting TYPE=nfs))
...because filesystem really does mean the whole FHS tree has been
mounted, and that the contents of /var are real (not just a stub
mountpoint). This won't work for anyone who uses NFS for a mountpoint
within the FHS (even if it's not /var and not otherwise needed for
launching statd) and probably won't work if an NFS filesystem is listed
in /etc/fstab (?) - but shouldn't cause extra problems when using
nfsroot since the kernel's internal statd is used in that case (I
think?)
Better approach?
==
Ideally, we could write something like:
start on (mounted-final MOUNTPOINT=/var) and (started portmap or
mounting TYPE=nfs)
Where mounted-final MOUNTPOINT=path means that all necessary mounts
have been done to populate path with its real FHS contents, and the
boot process won't mount anything else on top.
This could be implemented in a practical way in mountall if we don't
attempt to make it universal--- i.e., we don't ensure that it works for
every possible path, but we do make it work for top-level directories
defined by the FHS. To emit these events, mountall's must parse the
whole fstab and then act appropriately on each mount:
* When path is mounted:
* emit mounted MOUNTPOINT=path
* for d in {each FHS top-level dir}:
if no explicit mount for d or a parent of d in fstab:
emit mounted MOUNTPOINT=d
General approach
==
The above feels a bit messy and fragile, and doesn't solve the general
problem of configuration-dependent job start preconditions. So, it might
be better to implement outside mountall, by extending upstart with some
extra flexibility for job start conditions. For example:
start on $eval(mounted-final /var) and (started portmap or mounting
TYPE=nfs)
...where $eval(command arguments) is some magic new upstart event
expression syntax which runs an arbitrary command or script and uses its
output as part of the event expression. [I'm not suggesting exactly
that syntax of course -- I admit it's pretty hideous ;P]
In our case, mounted-final path is some widget which returns the
event expression mounted MOUNTPOINT=x, where x is the deepest path
listed in fstab that is a parent of, or is equal to, path. This is
pretty easy to script up. So it really depends on whether upstart
can/should be extended to support this kind of thing.
Retry approach
==
Finally, it might be interesting to consider whether it makes sense to define
specific retry conditions for jobs. This makes allows us to do better at
retries than dumb polling. I don't remember exactly the pattern-matching
capabilities for event key values, but I can imagine something like:
retry on (mounted MOUNTPOINT=/var) or (mounted MOUNTPOINT=/var/*)
retry on (mounted MOUNTPOINT=/var(/.*)?) # if regex is supported?
It still feels a bit wrong though... statd may spuriously succeed to
start if the rootfs contains /var/lib/nfs, but the real /var is
subsquently mounted on top of it (maybe after statd was started).
If a retry feature is added, it would be wise to limit the maximum
number of retries (as for respawn) or the maximum time period over which
retries will be attempted.
Thoughts?
** Patch added: my conservative workaround
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+attachment/1578208/+files/statd.diff
--
mountall
[Bug 525154] Re: mountall for /var races with rpc.statd
Regarding #58, while I agree in principle with much of what you say the underlying problem on my systems, as described in the 2nd paragraph in #55 is that rpc.statd can enter a state where it seems to be working, but actually isn't. (The only way out is to force rpc.statd to restart.) Consequently it is hard to imagine what sort of upstart syntax could possibly be used to avoid the subsequent failed NFS mounts. This is almost by definition an rpc.statd bug since no matter what upstart might or might not have done to start rpc.statd incorrectly (too early for instance), rpc.statd should be able to determine if it is, or isn't, working properly, and exit with a failed status in the latter case. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
In another limitation of upstart, or at least upstart is most likely responsible, the recovery mode boot hangs when in the /etc/fstab shown in #55 either /dev/sda6 is corrupt or there are network problems and the NFS mounts are not available. I guess using Mandriva spoiled me, because there booting failsafe resulted in the init being bright enough to start only the barest minimum of the operating system. Mandriva (and presumably RedHat and CentOS) come up when networks are broken and partitions (other than /boot and /) are screwed up. Not so Ubuntu - it tries to mount everything in /etc/fstab even when single is passed as a kernel option. Moreover, due to upstart's parallel nature, it isn't entirely clear at what point it is locking - it might not even be directly after the failed mount warnings, that might indirectly break something else. Perhaps there is some other Ubuntu/upstart specific kernel option to invoke a more low level boot, but if there is, that raises the question why it was not employed by the recovery mode boot option Ubuntu created at installation. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I strongly agree with the comments above. Against considerable management pressure to run all Linux boxes as RHEL/CentOS, I've insisted on building half our systems on Ubuntu Server, notwithstanding the really poor decision to transition from init.d to upstart. But to encounter this sort of brokenness, affecting stuff as common to professional use as keeping /var in its own partition and using nfs mounts, and then see an utter lack of commitment from Cannonical about fixing this, is sad. In other places I've been in the minority claiming Ubuntu is also enterprise ready. Now I'm kicking myself. Solution? Please? I'm not just looking for a kludge. This needs a clean, complete, universal solution. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
After 5 more reboots the at method failed. More exploration showed that status statd could show start/running and rpcinfo -p | grep status could show both the tcp and udp ports, and a couple of wait seconds could be put in there, and it could be run from an at command and even then killall -SIGUSR1 mountall would sometimes fail to respond to the signal. So I figured that perhaps statd was in a funky state too. Reset all the /etc/init files to their original values and put in rc.local and saf2.sh (the next two attachments). Have now booted 10 consecutive times with NSF mounts connected once I login.Of these 8 still had the statd error messages, but at least this hack finally made them mount in the end. Here is what shows up in the message.log when this happens: Aug 26 09:54:30 saf04 logger: NFSGLITCH marunning [ root 391 1 0 09:54 ?00:00:00 mountall --daemon ] Aug 26 09:54:30 saf04 logger: NFSGLITCH use at to kill mountall Aug 26 09:54:31 saf04 logger: NFSGLITCH2 statd: CLAIMS to be in start/running, restart it anyway Aug 26 09:54:32 saf04 logger: NFSGLITCH2 marunning? [ root 391 1 1 09:54 ?00:00:00 mountall --daemon ] Aug 26 09:54:32 saf04 logger: NFSGLITCH2 trying sigusr1 on mountall from at job Aug 26 09:54:32 saf04 kernel: [ 20.298991] RPC: Registered udp transport module. Aug 26 09:54:32 saf04 kernel: [ 20.298994] RPC: Registered tcp transport module. Aug 26 09:54:32 saf04 kernel: [ 20.298995] RPC: Registered tcp NFSv4.1 backchannel transport module. Aug 26 09:54:33 saf04 logger: NFSGLITCH2 kstat 0 marunning STILL? [ ] -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
see previous coment ** Attachment added: rc.local https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+attachment/1520756/+files/rc.local -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
see previous comment ** Attachment added: saf2.sh https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154/+attachment/1520757/+files/saf2.sh -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Making the changes suggested in comment #8 by ody has completed solved the problem for me. My clients have /var mounted on a separate local partition and users home mounted over the network via NFS. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
The changes in #8 did not work for me. /var is not mounted, it is just part of /, so the upstart's mounted test cannot be applied. The strangest thing about this whole mess is that, at least in my hands, rpc.statd can be in a state where to all outward appearances it is running normally (status statd shows start/running and rpcinfo -p shows both ports), yet until it is restarted (server statd stop; server statd start) mountall will never respond to a SIGUSR1 from within an init script, or (sometimes) an at job; yet mountall will always respond to that signal from root in a terminal! Moreover, in this strange state if root in a terminal enters mount /mnt/safserver/u1 it will result in all NFS mounts being made, not just that one. Rarely I also see in /var/log/boot.log mount.nfs: DNS resolution failed for safserver: Name or service not known This is just wrong because nsswitch.conf has files first for hosts, and safserver is in /etc/hosts. Probably yet another race condition. This one is definitely not persistent since (even without my fix) logging in after a failed NFS mount DNS is always working. Just discovered that the order of the lines in /etc/fstab also seems to make a difference: This one fails the most (every one of 4 boots): proc/proc procnodev,noexec,nosuid 0 0 LABEL=root / ext3errors=remount-ro 0 1 LABEL=boot /boot ext3defaults0 2 LABEL=swap noneswapsw 0 0 /dev/fd0/media/floppy0 autorw,user,noauto,exec,utf8 0 0 safserver:/u4/pdb /mnt/safserver/pdb nfs ro,bg,hard,intr 0 0 safserver:/u1 /mnt/safserver/u1 nfs rw,bg,hard,intr 0 0 /dev/sda1 /mnt/windows/C ntfs-3g ro 0 0 /dev/sda6 /mnt/windows/D ntfs-3g defaults 0 0 This one fails the least (none of 4 boots): proc/proc procnodev,noexec,nosuid 0 0 LABEL=root / ext3errors=remount-ro 0 1 LABEL=boot /boot ext3defaults0 2 LABEL=swap noneswapsw 0 0 /dev/fd0/media/floppy0 autorw,user,noauto,exec,utf8 0 0 /dev/sda1 /mnt/windows/C ntfs-3g ro 0 0 /dev/sda6 /mnt/windows/D ntfs-3g defaults 0 0 safserver:/u4/pdb /mnt/safserver/pdb nfs ro,bg,hard,intr 0 0 safserver:/u1 /mnt/safserver/u1 nfs rw,bg,hard,intr 0 0 This one is in between (fails about 75% of the time): proc/proc procnodev,noexec,nosuid 0 0 LABEL=root / ext3errors=remount-ro 0 1 LABEL=boot /boot ext3defaults0 2 LABEL=swap noneswapsw 0 0 /dev/fd0/media/floppy0 autorw,user,noauto,exec,utf8 0 0 /dev/sda1 /mnt/windows/C ntfs-3g ro 0 0 safserver:/u4/pdb /mnt/safserver/pdb nfs ro,bg,hard,intr 0 0 safserver:/u1 /mnt/safserver/u1 nfs rw,bg,hard,intr 0 0 /dev/sda6 /mnt/windows/D ntfs-3g defaults 0 0 Suggests that the ntfs-3g mounts provide enough of a delay so that the race condition between statd starting and nfs mounting is overcome. (Mostly, surely in enough boots it would fail even in the best order.) Upstart has far too many mysteries and hidden variables for my taste! -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
On my system /var is not in a separate partition, but I still have the infamous statd error messages on nfs mounts. Ubuntu 10.04.1 LTS mountall 2.15 upstart.6.5-6 I tried a bunch of things so far and none have worked reliably to cure it: 1. put a sleep 2 in /etc/mountall-net.conf to allow time for rpc.statd to start. 2. put in two different tests in /etc/init/statd.conf to make sure statd was working before it left that script - neither worked. One was by Andrew Edmunds from bug 610863. Then this one, which verifies the expected ports are open: post-start script while [ true ] do pcount=$(rpcinfo -p 2/dev/null | grep -c ' status$' 2/dev/null) if [ $pcount == 2 ] then break; else sleep 0.1 fi done end script 3. added --debug to the kernel boot line in grub. With messages spewing every which way I sometimes see the statd warning messages but in 7 tries it never came up without the NFS volumes mounted. Of course it took much longer to boot this way, and it looks to a naive user like the system has failed, so it is not acceptable for a production system. 4. added to /etc/rc.local service statd start killall -SIGUSR1 mountall Amazingly I have since rebooted a couple of times with (just) the /etc/rc.local change in place and while it helped, it sometimes came up not only with the infamous statd error messages from the failed nfs mounts, but with mountall still running! That looks to me like upstart (which perhaps should have been named upchuck) blew up and NEVER EVEN RAN rc.local. Either that or mountall ignored the USR1 signal. Both are really appalling possibilities in software which is supposed to be used in production environments. Heck, for all I know both failures may be present. Assuming at least one of the fixes from (2) above worked as intended then mountall is starting before the statd.conf script should have allowed it to. At least, that's the case if the post-start script must complete as a precondition of a successful statd start. If that isn't true, then what stanza should be used instead? -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I think it is finally working, but what a mess. Observations; 1. An explicit nfs mount like: mount /mnt/server/directory will not work from within rc.local while mountall is still running as a daemon. 2. An explicit killall -SIGUSR1 mountall from within rc.local will be ignored by mountall. 3. An explict killall -9 mountall will kill mountall, but then init goes nuts and throws up one of these General error mounting filesystems A maintenance shell will now be started etc. It does this even after one has already logged in on the console! 4. An explicit killall -SIGUSR1 mountall from a console session will allow the NFS mount to proceed, but of course that can only be done by root, and so it isn't a general solution for the end users 5. An explicit killall -SIGUSR1 mountall from an at job will also allow NFS mounts to complete. 6. I have no idea why mountall is hanging around and not starting the NFS connection. This is not a race condition with statd, as it can be tested and shown to be running while mountall is still twiddling its metaphorical fingers. So my final solution, until somebody fixes mountall, is to add to /etc/rc.local set +e # else any failed grep causes an exit marunning=`ps -ef | grep mountall | grep -v grep` logger MATHOG marunning [ $marunning ] set -e if [ -n $marunning ] then logger MATHOG use at to kill mountall at -f /etc/saf2.sh now else logger MATHOG ma not running, no kill needed fi Create the at file saf2.sh like: cat /etc/saf2.sh EOD # (name isn't important) #!/bin/bash set +e #allow time for rc.local to exit, this should probably be shorter sleep 4 marunning=`ps -ef | grep mountall | grep -v grep` logger MATHOG2 marunning? [ $marunning ] logger MATHOG2 trying sigusr1 on mountall from at job killall -SIGUSR1 mountall kstat=$? marunning=`ps -ef | grep mountall | grep -v grep` logger MATHOG2 kstat $kstat marunning STILL? [ $marunning ] set -e EOD When this boots the nfs/statd messages still appear on the console, but it does mount the NFS volumes within a few seconds, and these messages show up in /var/log/messages: Aug 25 14:48:07 saf04 logger: MATHOG marunning [ root 400 1 0 14:48 ?00:00:00 mountall --daemon ] Aug 25 14:48:07 saf04 logger: MATHOG use at to kill mountall Aug 25 14:48:10 saf04 logger: MATHOG2 marunning? [ root 400 1 0 14:48 ?00:00:00 mountall --daemon ] Aug 25 14:48:10 saf04 logger: MATHOG2 trying sigusr1 on mountall from at job Aug 25 14:48:10 saf04 logger: MATHOG2 kstat 0 marunning STILL? [ root 400 1 0 14:48 ?00:00:00 mountall --daemon ] Aug 25 14:48:10 saf04 kernel: [ 20.345629] RPC: Registered udp transport module. Aug 25 14:48:10 saf04 kernel: [ 20.345632] RPC: Registered tcp transport module. Aug 25 14:48:10 saf04 kernel: [ 20.345633] RPC: Registered tcp NFSv4.1 backchannel transport module. That is, when it glitches mountall is still running in rc.local, and when that is detected the at job is sent off and that finally triggers the NFS mount. Obviously you can dispense with the MATHOG tags, that is just in there for my debugging. Oh yes, there was also a bug in at, so that it would not accept a job. The solution for that was to uninstall and reinstall at with: apt-get remove --purge at apt-get install ubuntu-standard I wish I could rest easy with this work around, but the man page for mountall says: This is a temporary tool until init(8) itself gains the necessary flexibility to perform this processing; you should not rely on its behaviour. Consequently I'm looking forward to all of this falling apart again when they finally get around to that. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
A lame workaround is to simply add this to /etc/rc.local (before the exit 0): /sbin/initctl start statd -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I also fully agree with Brian in #37, John in #44 and Matthias in #45. We also have such a problem; we also use a separate /var-partiton as usual. Until someone is able to identify a solution that doesn't have this disadvantage, there's nothing that can be done here. Lucid is an LTS release and as such should be stable for 2 years. stable does not mean usable for all proposed uses. I feel a little bit mocked. Until now I was very satisfied by Ubuntu, but this problem already has a special dimension! Why did Canonical introduce Upstart, when it does not work correct in normal use? When this problem is not solvable, Canonical should go back to the former init solution. I gladly spend a few seconds more for (correct) booting. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
And I would add to #44, #45 and #46 that Canonical can not honestly ignore the fact that LTS releases are certainly preferred by enterprise/production environments, for obvious reasons (well, my 1-penny maybe mislead guess). Jumping to so much untested/unreliable stuff in a LTS release just makes me lose the trust I had built for Ubuntu until now. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I'm not an expert but I tested this: Create a script startstatd and save it in /etc/init.d. The script: #! /bin/sh service statd start mount /192.168.0.31: ... #add directives for mounting nfs shares Then create a symlink to /etc/rcS.d, for example @S91startstatd. It seems to be working well. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
There is a mistake in my script, sorry. The slash before 192.168.0.31 should be omitted: mount 192.168.0.31: ... #add directives for mounting nfs shares Furthermore: chmod 755 /etc/init.d/startstatd -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Couldn't we imagine using the output of df -P /var/lib/nfs/ | tail -n
1 | awk '{print $6}' to adapt the /etc/init/statd.conf script auto-
magically during the 'nfs-common' package installation (iow. having the
package's postinst script add the 'mounted MOUNTPOINT=...' stanza when
appropriate) ?
See proposed attached patch.
** Patch added: nfs-utils.bug525154.patch
http://launchpadlibrarian.net/53165046/nfs-utils.bug525154.patch
--
mountall for /var races with rpc.statd
https://bugs.launchpad.net/bugs/525154
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Sorry, previous patch contained a typo. New one attached (and should also be more resilient to package updates). I rebuilt the nfs-common_1.2.0-4ubuntu4_amd64.deb package and installed it with the expected auto-magic modification. ** Patch added: nfs-utils.bug525154.patch http://launchpadlibrarian.net/53166377/nfs-utils.bug525154.patch -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I have to agree with Brian in #37. It's looking more and more like Canonical are not interested in the user-base. First they remove Sun Java and expect everyone to report the bugs in openjdk and now this fiasco. In all the years I have been administering UNIX/Linux systems, it has been regarded as best-practice to have /var (and, indeed, /usr) on its own filesystem. I would suggest that upstart is really not production quality and, as such, should not be used for stable releases. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
I fully agree with Brian in #37 and John in #44. Ubuntu boasts on http://www.ubuntu.com/server: Ubuntu Server delivers services reliably, predictably and economically – and it easily integrates with your existing infrastructure. This is not true with regard to this bug. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Same here: this is making my MythTV frontend unusable as about half the time it can't see the MythTV NFS directories after booting. This used to work OK in 9.10; I upgraded it to Lucid in the hope that the newer kernel might fix USB remote bugs and instead I got an unusable network and lost most of the sound from xbmc. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
/var on separate filesystem, remote dir moounted using autofs. #13 works for me. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
So, what will be done about this (and the host of other mounting bugs) in Lucid? Lucid is an LTS release and as such should be stable for 2 years. It is not. This bug (and the host of other mounting bugs) make it not so. Yet no updates have come forth to solve the problem nor has there been any activity from Ubuntu developers. Have Ubuntu abandoned this (and the host of other mounting bugs) as simply too difficult to deal with? If not, please, let us know what the path forward, to a stable LTS (with separate /var, and/or NFS rooting, etc.) release is. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
As I wrote in comment #15, we don't have a viable workaround yet that doesn't inroduce other hangs / failures in other scenarios. A fix that will break all ability to further upgrade the system is worse than the status quo, because it means security fixes can't be applied. Until someone is able to identify a solution that doesn't have this disadvantage, there's nothing that can be done here. Lucid is an LTS release and as such should be stable for 2 years. stable does not mean usable for all proposed uses. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
On Tue, 2010-07-27 at 15:28 +, Steve Langasek wrote: As I wrote in comment #15, we don't have a viable workaround yet that doesn't inroduce other hangs / failures in other scenarios. I guess the question then is whether any effort is being spent towards such a solution. A fix that will break all ability to further upgrade the system is worse than the status quo, because it means security fixes can't be applied. Fair enough. But leaving systems unbootable for a portion of the users surely cannot be an acceptable solution either, yes? Until someone is able to identify a solution that doesn't have this disadvantage, there's nothing that can be done here. Who would this someone be? It's sounding an awful lot like nobody is actually working on the real root cause (and solution) of this issue and everyone is just hoping that somebody else will come up with a solution. stable does not mean usable for all proposed uses. So having a /var on a separate filesystem is fringe enough that those users should not be able to experience a stable system? /var on it's own filesystem is the only responsible way to manage a system. You can glom any other crap you want onto / but leaving /var on / to grow until it fills up / is simply irresponsible for any use-case except single-user machines. Is this single-user use case all that Ubuntu is interested in satisfying? Multi-user/server (i.e. corporate users) installations are not a useful user-base for Canonical? -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
This bug is affecting me too. It would be great to have a definitively solution. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
This seems to be the cause of both bugs 573919 and 590570. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
having /var on a separate filesystem, patch in #13 works for me too. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Description changed: Binary package hint: upstart - If one has /var (or /var/lib or /var/lib/nfs for that matter) on it's - own filesystem the statd.conf start races with the mounting of /var as + If one has /var (or /var/lib or /var/lib/nfs for that matter) on its own + filesystem the statd.conf start races with the mounting of /var as rpc.statd needs /var/lib/nfs to be available in order to work. I am sure this is not the only occurrence of this type of problem. A knee-jerk solution is to simply spin in statd.conf waiting for /var/lib/nfs to be available, but polling sucks, especially for something like upstart who's whole purpose is to be an event driven action manager. ** Description changed: Binary package hint: upstart If one has /var (or /var/lib or /var/lib/nfs for that matter) on its own filesystem the statd.conf start races with the mounting of /var as rpc.statd needs /var/lib/nfs to be available in order to work. I am sure this is not the only occurrence of this type of problem. A knee-jerk solution is to simply spin in statd.conf waiting for /var/lib/nfs to be available, but polling sucks, especially for - something like upstart who's whole purpose is to be an event driven + something like upstart whose whole purpose is to be an event driven action manager. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Thank you all for providing descriptions and solutions. My systems were upgraded from 9.10 where everything was working OK, and I was about to give up on Ubuntu when the systems didn't boot after upgrade. computer 1: /var on separate file system, mounting /home from nfs, hangs consistently on boot, fixed by removing fstab entry and mounting /home in /etc/rc.local computer 2: hmm, this is a special one... On one hand it is a perfectly normal dual-boot laptop, running Windows 7 and Ubuntu. On the other hand, that same Ubuntu partition is the disk of a virtual machine, so that I can run the same Ubuntu installation from within VirtualBox. Working OK when booting from BIOS, but was failing to boot from within VirtualBox until i moved the mount of a vboxsf share to rc.local. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Same problem here, /var on separate file system, patch in #13 seems to work. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
** Tags added: patch -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
On Mon, 2010-05-17 at 23:09 +, Steve Langasek wrote: In this reproducible case, is this the problem that the network has been brought up before /var is mounted? I.e., if you run 'killall -USR1 mountall' /instead of/ running a mount command by hand, does the boot continue? Yes sir, it does! Nice catch. Now, what's the fix? :-) -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
FWIW, this new 100% reproducible case is 100% reproducible for a reason, which I will get to in a minute. So, given that in two cases where I have had a stalled boot, signalling mountall with a USR1 has caused the boot to proceed. This seems like a race somewhere. The reason I was able to reproduce this 100% of the time on one particular given platform is because it's a PXE (i.e. netboot/netroot) system, booting from the network and mounting it's entire root (which includes /usr and /var) from an NFS server. In this scenario I found that allowing ifup to configure an interface that has already been configured by the kernel during the netboot was causing the system to hang. As a short-term solution until I could research the real, long-term solution, I decided that given that the interface was already up even before init was even started, I would just disable it in /etc/network/interfaces. That of course also prevents the initctl emit -n net-device-up ... in /etc/network/if-up.d/upstart from firing. But it would only prevent it from firing for the ethernet interface. Shouldn't it still fire for lo being ifup'd, giving mountall the USR1 it needs? -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
Oh, for NFS root, there's bug #537133. It seems there are known problems still with mountall in that configuration. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
On Wed, 2010-05-19 at 15:13 +, Steve Langasek wrote: Oh, for NFS root, there's bug #537133. It seems there are known problems still with mountall in that configuration. Funny enough, once I stopped dhclient from resetting an interface's address to 0.0.0.0 (and filing a bug upstream about it) and then re-enabled the interface in /etc/network/interfaces, my netboot/nfsroot system is working just peachy. Well apart from the ugly messages about failing to mount (other, non /) NFS filesystems first time through. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 525154] Re: mountall for /var races with rpc.statd
On Mon, 2010-05-17 at 23:09 +, Steve Langasek wrote: In this reproducible case, is this the problem that the network has been brought up before /var is mounted? I have not been able to test on that platform yet, however... I.e., if you run 'killall -USR1 mountall' /instead of/ running a mount command by hand, does the boot continue? On another platform that I was debugging last night, this indeed did seem to work. Hopefully I can get to my 100% reproducible case today and will let you know. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
So, I now have a case where this race prevents a successful boot completely, 100% of the time. The only way I can get a normal boot is to manually mount an NFS filesystem in another window while mountall/upstart has stalled. To clarify, the only way I can get this machine to boot is to boot the kernel with init=/bin/bash. Once I have the init/bash i open a vt with open -c 12 /bin/bash. I then exec init with exec /sbin/init and normal boot continues until upstart/mountall gets stuck waiting for NFS filesystems to mount, which never do. This will wait here forever if I don't intervene. To get the boot to continue, I switch to the bash I started on vt 12 and just mount one of the several nfs filesystems mountall is waiting for with: # mount /autohome/brian for example. At this point upstart resumes starting services and regular boot completes. Happy to provide any information needed to progress this issue to resolution. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
In this reproducible case, is this the problem that the network has been brought up before /var is mounted? I.e., if you run 'killall -USR1 mountall' /instead of/ running a mount command by hand, does the boot continue? -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
As an alternative hack/solution to this race (until it's resolved more elegantly within upstart), could we not simply spin in the statd.conf script waiting for /var/lib/nfs to be available? This would be most interesting to do because, in fact, I believe that /var/lib/nfs not being available when statd.conf runs is not the only issue that is causing rpc.statd startup to fail. I see failures reported by upstart, during boot, even after /var is mounted. Let me put such a spin lock in place and see how that goes. -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 525154] Re: mountall for /var races with rpc.statd
OK. The spin loop/lock seems to work much better than the start triggers. I still find however that mountall is trying to mount nfs filesystems before rpc.statd is started. Do we consider this an nfs-utils bug or an upstart/mountall bug? -- mountall for /var races with rpc.statd https://bugs.launchpad.net/bugs/525154 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
