Hi Carl, Glad to hear you got it all working in the end. Thanks for the info. It could end up saving us a lot of time (and others I'm sure too).
Many thanks! Mark On 16/01/13 00:35, Carl Schmidtmann wrote: > The problem turned out to be some garbage in the nscd cache. When the mux did > a user and group lookup it got back an invalid string pointer and then > crashed which took out all currently running jobs. Not a great failure mode > and why a nameserver call should ever return a bad pointer is beyond me. > > We use ldap and apparently once nscd gets garbage in its cache it stays there > until you invalidate the cache. Just restarting ncsd didn't fix it. To find > this it took getting the core dump to IBM which required changing the > permissions of / to 777 on the service node. IBM now knows of the problem > with dumping a core file and hopefully will fix it in a later release. IBM > also pointed us to a diag util that they supply to do a name/group lookup the > same way as there software to check the results. > > Because of this and other issues with nscd we really hate nscd. > > Carl > > ----- Original Message ----- >> Hi Carl, >> >> We're thinking about doing the upgrade to V1R2M0 next week, so >> naturally >> we're pretty interested in this. >> >> Hate to ask a very basic question, but have you added slurm to the >> list >> of users in the bgagent init script? For V1R1M2, we did the >> following: >> sed -e 's/OPTIONS="--users bgqadmin,bgws,bgqsysdb --workingdir >> $workingdir"/OPTIONS="--users bgqadmin,bgws,bgqsysdb,slurm >> --workingdir >> $workingdir"/' /etc/init.d/bgagent.orig> /etc/init.d/bgagent >> >> Thanks! >> Mark >> >> On 15/01/13 02:30, Carl Schmidtmann wrote: >>> >>> Is anyone else running slurm on a BG/Q and updated to BGQ SW >>> V1R2M0? We are seeing a problem where only users in the bgqadmin >>> group are able to run jobs. Anyone else trying to run a job causes >>> the runjob_mux process to core dump which takes out all running >>> jobs. So apparently there is some ownership or permissions changed >>> with the latest BGQ software that is incompatible with the way the >>> slurm runjob_mux module works. >>> >>> Has anyone else seen this and do you have a fix? IBM is working on >>> it but I thought if someone else has seen this we might get a >>> solution faster. >>> >>> Thanks, >>> Carl >> >
