I have been struggling with a gmond process on one cluster here that after some indeterminate period of time marks everything else in the cluster as down so instead of having a clustersize (as indicqated from the ganglia python command line client) of 135, the clustersize is 1. I have on the advice of Richard looked at the TN verses TMAX values in the XML that I get from this host when I telnet to it's port 8649 and the TN values are much bigger than the TMAX values. Richard said this indicated a problem but I am not sure where to go next with trying to diagnose this issue. I have ganglia running on other clusters just fine but this one cluster running 3.0.2 on RHEL4u2 seems to be having an issue.
Any suggestions as to what I can do to continue diagnosing the issue? -----Original Message----- >From: "Steven A. DuChene" <[EMAIL PROTECTED]> >Sent: Mar 28, 2006 5:48 PM >To: [EMAIL PROTECTED], [email protected] >Subject: RE: [Ganglia-general] gmond stops recognizing the rest of the cluster > >Yes, I have confirmed that when this condition occurs the TN figures are MUCH >greater >than the TMAX figures. I have double checked the routes and stuff is still >there (i.e. >netstat returns: > >Destination Gateway Genmask Flags MSS Window irtt Iface >239.2.11.71 0.0.0.0 255.255.255.255 UH 0 0 0 eth0 > >Where the host route for 239.2.11.71 is indeed still associated with the >interface >on the internal cluster network. > >Any suggestions then? > >-----Original Message----- >>From: [EMAIL PROTECTED] >>Sent: Mar 22, 2006 4:00 AM >>To: [EMAIL PROTECTED], [email protected] >>Subject: RE: [Ganglia-general] gmond stops recognizing the rest of the cluster >> >>Steven, >> >>if the problem is routing or actual packet loss, then that should be >>reflected >>by the XML output of the master gmond - the "down" host will have a TN >>(much) greater >>than the TMAX. e.g.: >> >><HOST NAME="ldndsm030000185.intranet.barcapint.com" IP="10.68.90.10" >>REPORTED="1143022788" TN="145" TMAX="20" DMAX="0" LOCATION="unspecified" >>GMOND_STARTED="1142870107"> >> >>There is also a very small chance that what you are seeing is related to >>the "Possible bug in hosts up calculation" thread. This bug causes an >>erroneous >>tagging of a data source as "old", which then changes the host up >>calculation to be >>one based on the wall clock of the gmetad server. Unless all the clocks >>are right, >>the host_up calculation is wrong. >> >> >>You may try this patch: >>http://sourceforge.net/mailarchive/message.php?msg_id=15170774 >> >>hope springs eternal, anyway. Myself, I only encountered the problem >>fixed here >>when I was federating clusters. >> >>There is also a host_up calculation in the PHP web stuff, ganglia.php, >>function host_alive. >>You could put debugging in there as well. >> >>kind regards, >>Richard >> >>-----Original Message----- >>From: [EMAIL PROTECTED] >>[mailto:[EMAIL PROTECTED] On Behalf Of Steven >>A. DuChene >>Sent: 21 March 2006 23:32 >>To: [email protected] >>Subject: [Ganglia-general] gmond stops recognizing the rest of the >>cluster >> >> >>I have a couple of mixed clusters here with AMD64/Opteron compute nodes >>and Intel EM64T Xeon managment nodes and I am running >>ganglia-gmond-3.0.2 >> >>Periodically (sometimes a couple or more times a day) I check the stats >>for the clusters and the cluster running RedHatEL-4.0 has a problem with >>the master gmond process (the one running on the management server with >>interfaces on the internal cluster network and the external lan here). >>It still responds to a query (using the python ganglia client or through >>the standard front end web >>page) but it stops seeing the client nodes and marks them off-line. It >>will indicate that only one host (itself) is actually up. I have to >>constantly be watching the outputs to see if this has happened and when >>it does do a: >> >> /etc/init.d/gmond restart >> >>That clears it up until next time. >> >>Any idea what could be causing this? I have been using ganglia to >>monitor clusters for quite some time but this is the first time i have >>seen the gmond process needing to be restarted to regain connection to >>the data stream running around inside the cluster. >> >>BTW, I have added a line to the /etc/init.d/gmond script to add a host >>route on the system with the dual network interfaces to point >>239.2.11.71 to the network interface that faces to the internal network >>of the cluster. >> >>I do not seem to have this issue with the cluster that has RedHatEL3 >>installed (same hardware thought). It is a smaller cluster (64 nodes >>verses the 128 cluster) though. >>-- >>Steven A. DuChene
