Hi Micheal and Rajiv. > Two questions spring to mind: > 1) What kind of load are you running under? It sounds like it could > be an out of control process (probably lam or ssh) which is flooding > itself and locking up communication. This came to mind first, along with > 2) Is there anything in your Crontab that fires at those times? The clearest observation was the very regular frequency of the lockups. In our case, it was almost 7 days to the hour for months. This made me think immediately of a cron or network activity, but we isolated the machine and stopped all cron jobs (for all users), and the problem persisted (we didn't let it go like that for more than one week, since the cluster is used heavily, so I can't comment on the regularity in that case). As for load levels, they were well below maximum and the lockup occured irrespective of the load. We did manage to trigger the lockup under large loads (on the disks and CPU's, mind you, not network), which may or may not mean something.
> Also, are you using openPBS/Maui/Lam-MPI or some other combination of > availible oscar packages? We were using PBSPro for many months and recently moved to TORQUE+MAUI. > Are there any users putting in scheduled jobs for those times? The lockups occurred irrespective of the queue state, not to mention job states. > Do you have any idea what kinds of jobs are inthe queue just before this happens? > openPbs in particular is quite buggy and the more complex a queue > setup is, the more likely it is to freak out. From watching the > torque/pbs and maui lists, they do strange things when they die. Our problem is very unlikely to be related to PBS, since as I said, we used PBSPro and sometimes the queue was empty or even stopped. We have tried to trigger the lockup in a reproduceable way, without success. All we know is that it is has a very nearly constant frequency over periods much longer than the average job time... Any more ideas ? Thanks bruce -- Bruce Becker UCT-CERN Research Center - University of Cape Town Private Bag RONDEBOSCH 7700 tel :(w) +27 21 650 3356 | (m) +27 82 537 9425 | (f) +27 21 650 3342 IM : AIM/Jabber - brucellino | Yahoo - uctbruce WEB :http://hep.phy.uct.ac.za/~becker "Viel hilft viel" ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Oscar-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/oscar-users
