Two questions spring to mind: 1) What kind of load are you running under? It sounds like it could be an out of control process (probably lam or ssh) which is flooding itself and locking up communication.
2) Is there anything in your Crontab that fires at those times? Also, are you using openPBS/Maui/Lam-MPI or some other combination of availible oscar packages? Are there any users putting in scheduled jobs for those times? Do you have any idea what kinds of jobs are in the queue just before this happens? openPbs in particular is quite buggy and the more complex a queue setup is, the more likely it is to freak out. From watching the torque/pbs and maui lists, they do strange things when they die. On Wed, 29 Sep 2004 11:06:45 +0200 (SAST), Bruce Becker <[EMAIL PROTECTED]> wrote: > Hi Rajiv. > > No help for you except to say that we have experienced a very similar > problem. Our head node hangs with a frequency of about 7 days (used to be > always on a friday afternoon, and became known as the beer-time-bug) > > We investigated every possible aspect of the system, which is running > dual PIII 1GHz CPU > OSCAR 3.0 > RH9 with custom 2.4.25 kernel > D-LINK GBE card. > ext3 fs > > > Cluster hangs for the frequency of 2 days, while trying to access the > > master node from any client node there is no response , ssh is not > > working . But if we ping the master node internal and lan IP we > > are getting reply .but unable to access the master node . then we > > are forced to shutdown all the cluster node and restart the cluster . > > Then it is working for 2 days continuously , then the same problem > > occurs. > We get exactly the same problem, except our frequency is 7 days. It wasn't > the network card, since we tried many versions and adaptations of the > driver and even changed the network card and tried it all again. It wasn't > hacked because we made a clean install and secured the system beyond > practical and the problem persisted. It wasn't unauthorised local access. > It wasn't the powe supply of the CPU overheating since we monitored those. > > I have no clue what it is and am resigned to living in hell. If anyone can > shed light on the problem, or suggest other diagnostics, I would love to > hear it ! > Bruce > -- > Bruce Becker > UCT-CERN Research Center - University of Cape Town > Private Bag RONDEBOSCH 7700 > tel :(w) +27 21 650 3356 | (m) +27 82 537 9425 | (f) +27 21 650 3342 > IM : AIM/Jabber - brucellino | Yahoo - uctbruce > WEB :http://hep.phy.uct.ac.za/~becker > "Viel hilft viel" > > > > > ------------------------------------------------------- > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal > Use IT products in your business? Tell us what you think of them. Give us > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more > http://productguide.itmanagersjournal.com/guidepromo.tmpl > _______________________________________________ > Oscar-users mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/oscar-users > ------------------------------------------------------- This SF.net email is sponsored by: IT Product Guide on ITManagersJournal Use IT products in your business? Tell us what you think of them. Give us Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more http://productguide.itmanagersjournal.com/guidepromo.tmpl _______________________________________________ Oscar-users mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/oscar-users
