Yes, I have confirmed that when this condition occurs the TN figures are MUCH greater than the TMAX figures. I have double checked the routes and stuff is still there (i.e. netstat returns:
Destination Gateway Genmask Flags MSS Window irtt Iface 239.2.11.71 0.0.0.0 255.255.255.255 UH 0 0 0 eth0 Where the host route for 239.2.11.71 is indeed still associated with the interface on the internal cluster network. Any suggestions then? -----Original Message----- >From: [EMAIL PROTECTED] >Sent: Mar 22, 2006 4:00 AM >To: [EMAIL PROTECTED], [email protected] >Subject: RE: [Ganglia-general] gmond stops recognizing the rest of the cluster > >Steven, > >if the problem is routing or actual packet loss, then that should be >reflected >by the XML output of the master gmond - the "down" host will have a TN >(much) greater >than the TMAX. e.g.: > ><HOST NAME="ldndsm030000185.intranet.barcapint.com" IP="10.68.90.10" >REPORTED="1143022788" TN="145" TMAX="20" DMAX="0" LOCATION="unspecified" >GMOND_STARTED="1142870107"> > >There is also a very small chance that what you are seeing is related to >the "Possible bug in hosts up calculation" thread. This bug causes an >erroneous >tagging of a data source as "old", which then changes the host up >calculation to be >one based on the wall clock of the gmetad server. Unless all the clocks >are right, >the host_up calculation is wrong. > > >You may try this patch: >http://sourceforge.net/mailarchive/message.php?msg_id=15170774 > >hope springs eternal, anyway. Myself, I only encountered the problem >fixed here >when I was federating clusters. > >There is also a host_up calculation in the PHP web stuff, ganglia.php, >function host_alive. >You could put debugging in there as well. > >kind regards, >Richard > >-----Original Message----- >From: [EMAIL PROTECTED] >[mailto:[EMAIL PROTECTED] On Behalf Of Steven >A. DuChene >Sent: 21 March 2006 23:32 >To: [email protected] >Subject: [Ganglia-general] gmond stops recognizing the rest of the >cluster > > >I have a couple of mixed clusters here with AMD64/Opteron compute nodes >and Intel EM64T Xeon managment nodes and I am running >ganglia-gmond-3.0.2 > >Periodically (sometimes a couple or more times a day) I check the stats >for the clusters and the cluster running RedHatEL-4.0 has a problem with >the master gmond process (the one running on the management server with >interfaces on the internal cluster network and the external lan here). >It still responds to a query (using the python ganglia client or through >the standard front end web >page) but it stops seeing the client nodes and marks them off-line. It >will indicate that only one host (itself) is actually up. I have to >constantly be watching the outputs to see if this has happened and when >it does do a: > > /etc/init.d/gmond restart > >That clears it up until next time. > >Any idea what could be causing this? I have been using ganglia to >monitor clusters for quite some time but this is the first time i have >seen the gmond process needing to be restarted to regain connection to >the data stream running around inside the cluster. > >BTW, I have added a line to the /etc/init.d/gmond script to add a host >route on the system with the dual network interfaces to point >239.2.11.71 to the network interface that faces to the internal network >of the cluster. > >I do not seem to have this issue with the cluster that has RedHatEL3 >installed (same hardware thought). It is a smaller cluster (64 nodes >verses the 128 cluster) though. >-- >Steven A. DuChene > > > > >------------------------------------------------------- >This SF.Net email is sponsored by xPML, a groundbreaking scripting >language that extends applications into web and mobile media. Attend the >live webcast and join the prime developer group breaking into this new >coding territory! >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >_______________________________________________ >Ganglia-general mailing list [email protected] >https://lists.sourceforge.net/lists/listinfo/ganglia-general > > >------------------------------------------------------------------------ >For more information about Barclays Capital, please >visit our web site at http://www.barcap.com. > > >Internet communications are not secure and therefore the Barclays >Group does not accept legal responsibility for the contents of this >message. Although the Barclays Group operates anti-virus programmes, >it does not accept responsibility for any damage whatsoever that is >caused by viruses being passed. Any views or opinions presented are >solely those of the author and do not necessarily represent those of the >Barclays Group. Replies to this email may be monitored by the Barclays >Group for operational or business reasons. > >------------------------------------------------------------------------ >
