The gmetad host is completely seperate from this cluster as I have one gmetad system outside the cluster that agrigates the gmond stuff from multiple clusters. The other clusters are just fine. Those other clusters are running RHEL3 while the problem one is running RHEL4. The gmond on the problem cluster headnode does not die. It just gets into a state where it is not reporting the other nodes in the cluster as alive. The way I double check this outside of the web page display from the gmetad system is I run the ganglia python command line client directly on the RHEL4 head node exhibiting the problem. The head node on the problem cluster has to have the gmond process stopped and restarted to regain the complete cluster in ganglia.
I am using multicasting and the head node does have multiple NICs but none are teamed. I also see this same issue on other non-compute nodes inside the cluster but since those are not the ones the outside gmetad system is pulling stats from it does not seem to affect anything. I.E. when I restart the gmond process on the headnode, any other systems exhibiting the problem (as seen from running the ganglia python client on that other system) report and are shown normally in the ganglia web and python clients. Because of the 2.6 kernel on the RHEL4 cluster where this problem is occuring should I try taking out the static host route to 239.2.11.71 to direct the ganglia packets to the correct network interface? I have always added this static route bit based on the information in the ganglia FAQ regarding systems with multiple network interfaces. -----Original Message----- >From: Martin Knoblauch <[EMAIL PROTECTED]> >Sent: Mar 29, 2006 11:59 PM >To: "Steven A. DuChene" <[EMAIL PROTECTED]>, >[email protected] >Subject: Re: [Ganglia-general] gmond unreliable on one cluster, must be >constantly restarted > >Steven, > > do you see anything in the /var/log/messages of the gmetad host? Do >you insert any custom metrics via "gmetric" or other means? > >Cheers >Martin > >--- "Steven A. DuChene" <[EMAIL PROTECTED]> wrote: > >> I have been struggling with a gmond process on one cluster here that >> after some indeterminate period of time marks everything else in the >> cluster as down so instead of having a clustersize (as indicqated >> from the >> ganglia python command line client) of 135, the clustersize is 1. >> I have on the advice of Richard looked at the TN verses TMAX values >> in the XML that I get from this host when I telnet to it's port 8649 >> and the TN values are much bigger than the TMAX values. Richard said >> this indicated a problem but I am not sure where to go next with >> trying >> to diagnose this issue. I have ganglia running on other clusters just >> fine >> but this one cluster running 3.0.2 on RHEL4u2 seems to be having an >> issue. >> >> Any suggestions as to what I can do to continue diagnosing the issue? >> >> -----Original Message----- >> >From: "Steven A. DuChene" <[EMAIL PROTECTED]> >> >Sent: Mar 28, 2006 5:48 PM >> >To: [EMAIL PROTECTED], >> [email protected] >> >Subject: RE: [Ganglia-general] gmond stops recognizing the rest of >> the cluster >> > >> >Yes, I have confirmed that when this condition occurs the TN figures >> are MUCH greater >> >than the TMAX figures. I have double checked the routes and stuff is >> still there (i.e. >> >netstat returns: >> > >> >Destination Gateway Genmask Flags MSS Window >> irtt Iface >> >239.2.11.71 0.0.0.0 255.255.255.255 UH 0 0 >> 0 eth0 >> > >> >Where the host route for 239.2.11.71 is indeed still associated with >> the interface >> >on the internal cluster network. >> > >> >Any suggestions then? >> > >> >-----Original Message----- >> >>From: [EMAIL PROTECTED] >> >>Sent: Mar 22, 2006 4:00 AM >> >>To: [EMAIL PROTECTED], >> [email protected] >> >>Subject: RE: [Ganglia-general] gmond stops recognizing the rest of >> the cluster >> >> >> >>Steven, >> >> >> >>if the problem is routing or actual packet loss, then that should >> be >> >>reflected >> >>by the XML output of the master gmond - the "down" host will have a >> TN >> >>(much) greater >> >>than the TMAX. e.g.: >> >> >> >><HOST NAME="ldndsm030000185.intranet.barcapint.com" >> IP="10.68.90.10" >> >>REPORTED="1143022788" TN="145" TMAX="20" DMAX="0" >> LOCATION="unspecified" >> >>GMOND_STARTED="1142870107"> >> >> >> >>There is also a very small chance that what you are seeing is >> related to >> >>the "Possible bug in hosts up calculation" thread. This bug causes >> an >> >>erroneous >> >>tagging of a data source as "old", which then changes the host up >> >>calculation to be >> >>one based on the wall clock of the gmetad server. Unless all the >> clocks >> >>are right, >> >>the host_up calculation is wrong. >> >> >> >> >> >>You may try this patch: >> >>http://sourceforge.net/mailarchive/message.php?msg_id=15170774 >> >> >> >>hope springs eternal, anyway. Myself, I only encountered the >> problem >> >>fixed here >> >>when I was federating clusters. >> >> >> >>There is also a host_up calculation in the PHP web stuff, >> ganglia.php, >> >>function host_alive. >> >>You could put debugging in there as well. >> >> >> >>kind regards, >> >>Richard >> >> >> >>-----Original Message----- >> >>From: [EMAIL PROTECTED] >> >>[mailto:[EMAIL PROTECTED] On Behalf Of >> Steven >> >>A. DuChene >> >>Sent: 21 March 2006 23:32 >> >>To: [email protected] >> >>Subject: [Ganglia-general] gmond stops recognizing the rest of the >> >>cluster >> >> >> >> >> >>I have a couple of mixed clusters here with AMD64/Opteron compute >> nodes >> >>and Intel EM64T Xeon managment nodes and I am running >> >>ganglia-gmond-3.0.2 >> >> >> >>Periodically (sometimes a couple or more times a day) I check the >> stats >> >>for the clusters and the cluster running RedHatEL-4.0 has a problem >> with >> >>the master gmond process (the one running on the management server >> with >> >>interfaces on the internal cluster network and the external lan >> here). >> >>It still responds to a query (using the python ganglia client or >> through >> >>the standard front end web >> >>page) but it stops seeing the client nodes and marks them off-line. >> It >> >>will indicate that only one host (itself) is actually up. I have to >> >>constantly be watching the outputs to see if this has happened and >> when >> >>it does do a: >> >> >> >> /etc/init.d/gmond restart >> >> >> >>That clears it up until next time. >> >> >> >>Any idea what could be causing this? I have been using ganglia to >> >>monitor clusters for quite some time but this is the first time i >> have >> >>seen the gmond process needing to be restarted to regain connection >> to >> >>the data stream running around inside the cluster. >> >> >> >>BTW, I have added a line to the /etc/init.d/gmond script to add a >> host >> >>route on the system with the dual network interfaces to point >> >>239.2.11.71 to the network interface that faces to the internal >> network >> >>of the cluster. >> >> >> >>I do not seem to have this issue with the cluster that has >> RedHatEL3 >> >>installed (same hardware thought). It is a smaller cluster (64 >> nodes >> >>verses the 128 cluster) though. >> >>-- >> >>Steven A. DuChene >> >> >> >> >> ------------------------------------------------------- >> This SF.Net email is sponsored by xPML, a groundbreaking scripting >> language >> that extends applications into web and mobile media. Attend the live >> webcast >> and join the prime developer group breaking into this new coding >> territory! >> >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >> _______________________________________________ >> Ganglia-general mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/ganglia-general >> >> > > >------------------------------------------------------ >Martin Knoblauch >email: k n o b i AT knobisoft DOT de >www: http://www.knobisoft.de > > >------------------------------------------------------- >This SF.Net email is sponsored by xPML, a groundbreaking scripting language >that extends applications into web and mobile media. Attend the live webcast >and join the prime developer group breaking into this new coding territory! >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >_______________________________________________ >Ganglia-general mailing list >[email protected] >https://lists.sourceforge.net/lists/listinfo/ganglia-general
