The problem turned out to be some garbage in the nscd cache. When the mux did a user and group lookup it got back an invalid string pointer and then crashed which took out all currently running jobs. Not a great failure mode and why a nameserver call should ever return a bad pointer is beyond me.
We use ldap and apparently once nscd gets garbage in its cache it stays there until you invalidate the cache. Just restarting ncsd didn't fix it. To find this it took getting the core dump to IBM which required changing the permissions of / to 777 on the service node. IBM now knows of the problem with dumping a core file and hopefully will fix it in a later release. IBM also pointed us to a diag util that they supply to do a name/group lookup the same way as there software to check the results. Because of this and other issues with nscd we really hate nscd. Carl ----- Original Message ----- > Hi Carl, > > We're thinking about doing the upgrade to V1R2M0 next week, so > naturally > we're pretty interested in this. > > Hate to ask a very basic question, but have you added slurm to the > list > of users in the bgagent init script? For V1R1M2, we did the > following: > sed -e 's/OPTIONS="--users bgqadmin,bgws,bgqsysdb --workingdir > $workingdir"/OPTIONS="--users bgqadmin,bgws,bgqsysdb,slurm > --workingdir > $workingdir"/' /etc/init.d/bgagent.orig > /etc/init.d/bgagent > > Thanks! > Mark > > On 15/01/13 02:30, Carl Schmidtmann wrote: > > > > Is anyone else running slurm on a BG/Q and updated to BGQ SW > > V1R2M0? We are seeing a problem where only users in the bgqadmin > > group are able to run jobs. Anyone else trying to run a job causes > > the runjob_mux process to core dump which takes out all running > > jobs. So apparently there is some ownership or permissions changed > > with the latest BGQ software that is incompatible with the way the > > slurm runjob_mux module works. > > > > Has anyone else seen this and do you have a fix? IBM is working on > > it but I thought if someone else has seen this we might get a > > solution faster. > > > > Thanks, > > Carl > -- Carl Schmidtmann Center for Integrated Research Computing University of Rochester
