Steven, if the problem is routing or actual packet loss, then that should be reflected by the XML output of the master gmond - the "down" host will have a TN (much) greater than the TMAX. e.g.:
<HOST NAME="ldndsm030000185.intranet.barcapint.com" IP="10.68.90.10" REPORTED="1143022788" TN="145" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1142870107"> There is also a very small chance that what you are seeing is related to the "Possible bug in hosts up calculation" thread. This bug causes an erroneous tagging of a data source as "old", which then changes the host up calculation to be one based on the wall clock of the gmetad server. Unless all the clocks are right, the host_up calculation is wrong. You may try this patch: http://sourceforge.net/mailarchive/message.php?msg_id=15170774 hope springs eternal, anyway. Myself, I only encountered the problem fixed here when I was federating clusters. There is also a host_up calculation in the PHP web stuff, ganglia.php, function host_alive. You could put debugging in there as well. kind regards, Richard -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steven A. DuChene Sent: 21 March 2006 23:32 To: [email protected] Subject: [Ganglia-general] gmond stops recognizing the rest of the cluster I have a couple of mixed clusters here with AMD64/Opteron compute nodes and Intel EM64T Xeon managment nodes and I am running ganglia-gmond-3.0.2 Periodically (sometimes a couple or more times a day) I check the stats for the clusters and the cluster running RedHatEL-4.0 has a problem with the master gmond process (the one running on the management server with interfaces on the internal cluster network and the external lan here). It still responds to a query (using the python ganglia client or through the standard front end web page) but it stops seeing the client nodes and marks them off-line. It will indicate that only one host (itself) is actually up. I have to constantly be watching the outputs to see if this has happened and when it does do a: /etc/init.d/gmond restart That clears it up until next time. Any idea what could be causing this? I have been using ganglia to monitor clusters for quite some time but this is the first time i have seen the gmond process needing to be restarted to regain connection to the data stream running around inside the cluster. BTW, I have added a line to the /etc/init.d/gmond script to add a host route on the system with the dual network interfaces to point 239.2.11.71 to the network interface that faces to the internal network of the cluster. I do not seem to have this issue with the cluster that has RedHatEL3 installed (same hardware thought). It is a smaller cluster (64 nodes verses the 128 cluster) though. -- Steven A. DuChene ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Ganglia-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ganglia-general ------------------------------------------------------------------------ For more information about Barclays Capital, please visit our web site at http://www.barcap.com. Internet communications are not secure and therefore the Barclays Group does not accept legal responsibility for the contents of this message. Although the Barclays Group operates anti-virus programmes, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Barclays Group. Replies to this email may be monitored by the Barclays Group for operational or business reasons. ------------------------------------------------------------------------
