Check your XML output from the gmond process serving the particular
cluster vs. the gmetad XML that feeds the PHP/rrd's, and then vs. the
individual gmond node outputs... I'm expecting you'd find that the data
is proper from the end client, and stops being fed correctly up to the
gmetad process when it queries your aggregator gmond for the cluster.
Also, run a tethereal and watch to make sure it's still receiving
{uni|multi}cast from the other nodes, and that it hasn't started
rejecting them.
That's where I'd start, curious to see where it goes.
/eli
Steven A. DuChene wrote:
I have a couple of mixed clusters here with AMD64/Opteron compute nodes
and Intel EM64T Xeon managment nodes and I am running ganglia-gmond-3.0.2
Periodically (sometimes a couple or more times a day) I check the stats for the
clusters and the cluster running RedHatEL-4.0 has a problem with the master
gmond process (the one running on the management server with interfaces on
the internal cluster network and the external lan here). It still responds to a
query (using the python ganglia client or through the standard front end web
page) but it stops seeing the client nodes and marks them off-line. It will
indicate
that only one host (itself) is actually up. I have to constantly be watching the
outputs to see if this has happened and when it does do a:
/etc/init.d/gmond restart
That clears it up until next time.
Any idea what could be causing this? I have been using ganglia to monitor
clusters
for quite some time but this is the first time i have seen the gmond process
needing
to be restarted to regain connection to the data stream running around inside
the cluster.
BTW, I have added a line to the /etc/init.d/gmond script to add a host route on
the
system with the dual network interfaces to point 239.2.11.71 to the network
interface
that faces to the internal network of the cluster.
I do not seem to have this issue with the cluster that has RedHatEL3 installed
(same
hardware thought). It is a smaller cluster (64 nodes verses the 128 cluster)
though.
--
Steven A. DuChene
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
