On Mar 3, 2015 8:49 AM, "P. Larry Nelson" <[email protected]> wrote: > > I am seeing a bizarre bug where an SL6.x system hangs on either > shutdown or reboot at the point where it wants to shutdown the > loopback interface. > > Let me start off by saying I'm running a mixed shop of SL5.x servers > (DNS, NIS, NTP, DHCP, NFS, etc.) along with a bunch of new cluster-esque > nodes running SL6.x. All new SL6 nodes are Dell R410, R510, R710, for > whatever that's worth, but I don't believe they have anything to do > with the bug, per se. > > Since building these new SL6 nodes many weeks back, they have all > exhibited this extremely annoying habit of hanging on shutdown or > reboot at the shutdown of the loopback interface. > Eventually (for the most part) they stop spinning whatever wheels > they're spinning and do manage to complete either the shutdown or > reboot, but it takes upwards of 15, 20, or 30 minutes! Usually > I can't wait that long and just do a power off/on of the node. > > No amount of trying to find out what they are doing has worked, > from trying to open another console window (Alt-F1, etc.) at > shutdown/reboot to having top running in one terminal window while > doing a 'service network restart' in another. Everything just freezes! > > I tried any number of things over the past several weeks, including > ripping out NetworkManager knowing that it has had a history of mucking > things up. No luck. They still hang. > > On another front, I was having some UID/GID problems with the mix of > NFS v3 from my SL5.x file servers and NFS v4 on the SL6 nodes, so > I forced all mounts to use NFS v3. I thought maybe that could be > the problem, but again, no luck - still hanging. > > Revisiting it again in earnest this weekend via Google, I came up > empty as all hits seemed to have something to do with scenarios that > just did not apply, including many hits about a problem with running > the iscsi daemon (and there was a patch for that). But I'm not running > the iscsi daemon. It's not even installed. > > One comment by someone who also had the same problem was that he, not > ever figuring out the cause, just commented out the line in > /etc/init.d/network that shuts down the loopback interface, saying it's > not a real device anyway, so what the hell. > > So yesterday I thought I'd try the commenting out the loopback shutdown tactic on a test system. Sure enough, the reboot was normal with no > hangs. > > Ok, at least now I have a workaround, though that seems pretty kludgy. > > I decided to try and nail the culprit down with a fresh rebuild of > a test system and see just where in the build process the bug appears. > > After the basic install of SL6, the system reboots just fine. > Then do a 'yum update' with all its hundreds of patches. > It reboots just fine, as I expected. > > So the first "local" change was to configure NIS. > Try the reboot. Reboots fine. > > [ok, here is where it becomes bizarre] > Modify /etc/nsswitch.conf to switch the order of "files nis" to > "nis files" for passwd, shadow, and group, as I've always done. > Reboot. Boom! It hangs at loopback interface shutdown! >
I want to thank you for giving all the details of your testing. I would like to use it as a future example of how to be constructive and helpful to other people needing help. So have you looked at nscd any? Does having nscd turned on or off alter this problem. Also what is in hosts and is the NIS server listed. Thanks > I repeated this many times to be sure, and it happens the same on > every SL6.x node. > > Bug or feature? I can't imagine it to be a feature nor can I > fathom what the order of "files" and "nis" in /etc/nsswitch.conf > has to do with the hanging of the loopback interface shutdown. > It's possible that an SL6.x NIS server might correct the situation, > but I have no time right now to spend a week on that not knowing > it would even work. > > Comments and suggestions are welcome. > > - Larry > > -- > P. Larry Nelson (217-244-9855) | IT Administrator > 461 Loomis Lab | High Energy Physics Group > 1110 W. Green St., Urbana, IL | Physics Dept., Univ. of Ill. > MailTo:[email protected] | http://www.brf-llc.com/lnelson/ > ------------------------------------------------------------------- > "Information without accountability is just noise." - P.L. Nelson
