Steven, do you see anything in the /var/log/messages of the gmetad host? Do you insert any custom metrics via "gmetric" or other means?
Cheers Martin --- "Steven A. DuChene" <[EMAIL PROTECTED]> wrote: > I have been struggling with a gmond process on one cluster here that > after some indeterminate period of time marks everything else in the > cluster as down so instead of having a clustersize (as indicqated > from the > ganglia python command line client) of 135, the clustersize is 1. > I have on the advice of Richard looked at the TN verses TMAX values > in the XML that I get from this host when I telnet to it's port 8649 > and the TN values are much bigger than the TMAX values. Richard said > this indicated a problem but I am not sure where to go next with > trying > to diagnose this issue. I have ganglia running on other clusters just > fine > but this one cluster running 3.0.2 on RHEL4u2 seems to be having an > issue. > > Any suggestions as to what I can do to continue diagnosing the issue? > > -----Original Message----- > >From: "Steven A. DuChene" <[EMAIL PROTECTED]> > >Sent: Mar 28, 2006 5:48 PM > >To: [EMAIL PROTECTED], > [email protected] > >Subject: RE: [Ganglia-general] gmond stops recognizing the rest of > the cluster > > > >Yes, I have confirmed that when this condition occurs the TN figures > are MUCH greater > >than the TMAX figures. I have double checked the routes and stuff is > still there (i.e. > >netstat returns: > > > >Destination Gateway Genmask Flags MSS Window > irtt Iface > >239.2.11.71 0.0.0.0 255.255.255.255 UH 0 0 > 0 eth0 > > > >Where the host route for 239.2.11.71 is indeed still associated with > the interface > >on the internal cluster network. > > > >Any suggestions then? > > > >-----Original Message----- > >>From: [EMAIL PROTECTED] > >>Sent: Mar 22, 2006 4:00 AM > >>To: [EMAIL PROTECTED], > [email protected] > >>Subject: RE: [Ganglia-general] gmond stops recognizing the rest of > the cluster > >> > >>Steven, > >> > >>if the problem is routing or actual packet loss, then that should > be > >>reflected > >>by the XML output of the master gmond - the "down" host will have a > TN > >>(much) greater > >>than the TMAX. e.g.: > >> > >><HOST NAME="ldndsm030000185.intranet.barcapint.com" > IP="10.68.90.10" > >>REPORTED="1143022788" TN="145" TMAX="20" DMAX="0" > LOCATION="unspecified" > >>GMOND_STARTED="1142870107"> > >> > >>There is also a very small chance that what you are seeing is > related to > >>the "Possible bug in hosts up calculation" thread. This bug causes > an > >>erroneous > >>tagging of a data source as "old", which then changes the host up > >>calculation to be > >>one based on the wall clock of the gmetad server. Unless all the > clocks > >>are right, > >>the host_up calculation is wrong. > >> > >> > >>You may try this patch: > >>http://sourceforge.net/mailarchive/message.php?msg_id=15170774 > >> > >>hope springs eternal, anyway. Myself, I only encountered the > problem > >>fixed here > >>when I was federating clusters. > >> > >>There is also a host_up calculation in the PHP web stuff, > ganglia.php, > >>function host_alive. > >>You could put debugging in there as well. > >> > >>kind regards, > >>Richard > >> > >>-----Original Message----- > >>From: [EMAIL PROTECTED] > >>[mailto:[EMAIL PROTECTED] On Behalf Of > Steven > >>A. DuChene > >>Sent: 21 March 2006 23:32 > >>To: [email protected] > >>Subject: [Ganglia-general] gmond stops recognizing the rest of the > >>cluster > >> > >> > >>I have a couple of mixed clusters here with AMD64/Opteron compute > nodes > >>and Intel EM64T Xeon managment nodes and I am running > >>ganglia-gmond-3.0.2 > >> > >>Periodically (sometimes a couple or more times a day) I check the > stats > >>for the clusters and the cluster running RedHatEL-4.0 has a problem > with > >>the master gmond process (the one running on the management server > with > >>interfaces on the internal cluster network and the external lan > here). > >>It still responds to a query (using the python ganglia client or > through > >>the standard front end web > >>page) but it stops seeing the client nodes and marks them off-line. > It > >>will indicate that only one host (itself) is actually up. I have to > >>constantly be watching the outputs to see if this has happened and > when > >>it does do a: > >> > >> /etc/init.d/gmond restart > >> > >>That clears it up until next time. > >> > >>Any idea what could be causing this? I have been using ganglia to > >>monitor clusters for quite some time but this is the first time i > have > >>seen the gmond process needing to be restarted to regain connection > to > >>the data stream running around inside the cluster. > >> > >>BTW, I have added a line to the /etc/init.d/gmond script to add a > host > >>route on the system with the dual network interfaces to point > >>239.2.11.71 to the network interface that faces to the internal > network > >>of the cluster. > >> > >>I do not seem to have this issue with the cluster that has > RedHatEL3 > >>installed (same hardware thought). It is a smaller cluster (64 > nodes > >>verses the 128 cluster) though. > >>-- > >>Steven A. DuChene > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Ganglia-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/ganglia-general > > ------------------------------------------------------ Martin Knoblauch email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
