Hey Bruce: Have you looked at relevant logs to see what was going on at the time of the system crash?
- PBS logs - /var/log/messages I would suggest you run all the updates available for RH9 since you have installed the system too. Cheers, Bernard > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Bruce Becker > Sent: Wednesday, September 29, 2004 6:19 > To: Michael Edwards > Cc: Rajiv Jaisankar; [EMAIL PROTECTED] > Subject: Re: [Oscar-users] Master Node hanging for the > frequency of 2 days > > Hi Micheal and Rajiv. > > > Two questions spring to mind: > > 1) What kind of load are you running under? It sounds like > it could > > be an out of control process (probably lam or ssh) which is > flooding > > itself and locking up communication. > This came to mind first, along with > > 2) Is there anything in your Crontab that fires at those times? > The clearest observation was the very regular frequency of > the lockups. In our case, it was almost 7 days to the hour > for months. This made me think immediately of a cron or > network activity, but we isolated the machine and stopped all > cron jobs (for all users), and the problem persisted (we > didn't let it go like that for more than one week, since the > cluster is used heavily, so I can't comment on the regularity > in that case). As for load levels, they were well below > maximum and the lockup occured irrespective of the load. We > did manage to trigger the lockup under large loads (on the > disks and CPU's, mind you, not network), which may or may not > mean something. > > > Also, are you using openPBS/Maui/Lam-MPI or some other > combination of > > availible oscar packages? > We were using PBSPro for many months and recently moved to > TORQUE+MAUI. > > Are there any users putting in scheduled jobs for those times? > The lockups occurred irrespective of the queue state, not to > mention job states. > > Do you have any idea what kinds of jobs are inthe queue > just before this happens? > > openPbs in particular is quite buggy and the more complex a queue > > setup is, the more likely it is to freak out. From watching the > > torque/pbs and maui lists, they do strange things when they die. > Our problem is very unlikely to be related to PBS, since as I > said, we used PBSPro and sometimes the queue was empty or > even stopped. > > We have tried to trigger the lockup in a reproduceable way, > without success. All we know is that it is has a very nearly > constant frequency over periods much longer than the average > job time... Any more ideas ? > > Thanks > bruce > > -- > Bruce Becker > UCT-CERN Research Center - University of Cape Town Private > Bag RONDEBOSCH 7700 tel :(w) +27 21 650 3356 | (m) +27 82 537 > 9425 | (f) +27 21 650 3342 IM : AIM/Jabber - brucellino | > Yahoo - uctbruce > WEB :http://hep.phy.uct.ac.za/~becker > "Viel hilft viel" > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: IT Product Guide on > ITManagersJournal Use IT products in your business? Tell us > what you think of them. Give us Your Opinions, Get Free > ThinkGeek Gift Certificates! Click to find out more > http://productguide.itmanagersjournal.com/guidepromo.tmpl > _______________________________________________ > Oscar-users mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/oscar-users > > ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Oscar-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/oscar-users
