There are a few simple and obvious steps. BTW, it is good that TN is greater than TMAX in some sense, because this means that gmetad and all the php stuff is not saying anything that is wrong wrt to the XML stream.
So have you done a simple tcpdump of UDP port 8649 on the headnode? Do the UDP packets arrive? I am also trying to remember whether you unicast or multicast. At least with unicast I pretty well know that anything of the right address arriving on the NIC will be seen by gmond. Consider unicast for debugging, although perhaps I am being unduly conservative. Also did you mention whether your headnode has multiple NICs? Are they teamed? - Richard -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Steven A. DuChene Sent: 30 March 2006 03:18 To: [email protected] Subject: [Ganglia-general] gmond unreliable on one cluster, must be constantly restarted I have been struggling with a gmond process on one cluster here that after some indeterminate period of time marks everything else in the cluster as down so instead of having a clustersize (as indicqated from the ganglia python command line client) of 135, the clustersize is 1. I have on the advice of Richard looked at the TN verses TMAX values in the XML that I get from this host when I telnet to it's port 8649 and the TN values are much bigger than the TMAX values. Richard said this indicated a problem but I am not sure where to go next with trying to diagnose this issue. I have ganglia running on other clusters just fine but this one cluster running 3.0.2 on RHEL4u2 seems to be having an issue. Any suggestions as to what I can do to continue diagnosing the issue? -----Original Message----- >From: "Steven A. DuChene" <[EMAIL PROTECTED]> >Sent: Mar 28, 2006 5:48 PM >To: [EMAIL PROTECTED], >[email protected] >Subject: RE: [Ganglia-general] gmond stops recognizing the rest of the cluster > >Yes, I have confirmed that when this condition occurs the TN figures >are MUCH greater than the TMAX figures. I have double checked the >routes and stuff is still there (i.e. netstat returns: > >Destination Gateway Genmask Flags MSS Window irtt Iface >239.2.11.71 0.0.0.0 255.255.255.255 UH 0 0 0 eth0 > >Where the host route for 239.2.11.71 is indeed still associated with >the interface on the internal cluster network. > >Any suggestions then? > >-----Original Message----- >>From: [EMAIL PROTECTED] >>Sent: Mar 22, 2006 4:00 AM >>To: [EMAIL PROTECTED], >>[email protected] >>Subject: RE: [Ganglia-general] gmond stops recognizing the rest of the cluster >> >>Steven, >> >>if the problem is routing or actual packet loss, then that should be >>reflected by the XML output of the master gmond - the "down" host will >>have a TN >>(much) greater >>than the TMAX. e.g.: >> >><HOST NAME="ldndsm030000185.intranet.barcapint.com" IP="10.68.90.10" >>>>REPORTED="1143022788" TN="145" TMAX="20" DMAX="0" >>LOCATION="unspecified" GMOND_STARTED="1142870107"> >> >>There is also a very small chance that what you are seeing is related >>to the "Possible bug in hosts up calculation" thread. This bug causes >>an erroneous tagging of a data source as "old", which then changes the >>host up calculation to be >>one based on the wall clock of the gmetad server. Unless all the clocks >>are right, >>the host_up calculation is wrong. >> >> >>You may try this patch: >>http://sourceforge.net/mailarchive/message.php?msg_id=15170774 >> >>hope springs eternal, anyway. Myself, I only encountered the problem >>fixed here when I was federating clusters. >> >>There is also a host_up calculation in the PHP web stuff, ganglia.php, >>function host_alive. You could put debugging in there as well. >> >>kind regards, >>Richard >> >>-----Original Message----- >>From: [EMAIL PROTECTED] >>[mailto:[EMAIL PROTECTED] On Behalf Of >>Steven A. DuChene >>Sent: 21 March 2006 23:32 >>To: [email protected] >>Subject: [Ganglia-general] gmond stops recognizing the rest of the >>cluster >> >> >>I have a couple of mixed clusters here with AMD64/Opteron compute >>nodes and Intel EM64T Xeon managment nodes and I am running >>ganglia-gmond-3.0.2 >> >>Periodically (sometimes a couple or more times a day) I check the >>stats for the clusters and the cluster running RedHatEL-4.0 has a >>problem with the master gmond process (the one running on the >>management server with interfaces on the internal cluster network and >>the external lan here). It still responds to a query (using the python >>ganglia client or through the standard front end web >>page) but it stops seeing the client nodes and marks them off-line. It >>will indicate that only one host (itself) is actually up. I have to >>constantly be watching the outputs to see if this has happened and >>when it does do a: >> >> /etc/init.d/gmond restart >> >>That clears it up until next time. >> >>Any idea what could be causing this? I have been using ganglia to >>monitor clusters for quite some time but this is the first time i have >>seen the gmond process needing to be restarted to regain connection to >>the data stream running around inside the cluster. >> >>BTW, I have added a line to the /etc/init.d/gmond script to add a host >>route on the system with the dual network interfaces to point >>239.2.11.71 to the network interface that faces to the internal >>network of the cluster. >> >>I do not seem to have this issue with the cluster that has RedHatEL3 >>installed (same hardware thought). It is a smaller cluster (64 nodes >>verses the 128 cluster) though. >>-- >>Steven A. DuChene ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Ganglia-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ganglia-general ------------------------------------------------------------------------ For more information about Barclays Capital, please visit our web site at http://www.barcap.com. Internet communications are not secure and therefore the Barclays Group does not accept legal responsibility for the contents of this message. Although the Barclays Group operates anti-virus programmes, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Barclays Group. Replies to this email may be monitored by the Barclays Group for operational or business reasons. ------------------------------------------------------------------------
