And seeing now that you add a route in the init script, that's your
probable culprit. Check your routing table when it fails vs. "good",
and I'm guessing you'll find something deleted your route...
/eli
Eli Stair wrote:
Check your XML output from the gmond process serving the particular
cluster vs. the gmetad XML that feeds the PHP/rrd's, and then vs. the
individual gmond node outputs... I'm expecting you'd find that the data
is proper from the end client, and stops being fed correctly up to the
gmetad process when it queries your aggregator gmond for the cluster.
Also, run a tethereal and watch to make sure it's still receiving
{uni|multi}cast from the other nodes, and that it hasn't started
rejecting them.
That's where I'd start, curious to see where it goes.
/eli
Steven A. DuChene wrote:
I have a couple of mixed clusters here with AMD64/Opteron compute nodes
and Intel EM64T Xeon managment nodes and I am running ganglia-gmond-3.0.2
Periodically (sometimes a couple or more times a day) I check the
stats for the
clusters and the cluster running RedHatEL-4.0 has a problem with the
master
gmond process (the one running on the management server with
interfaces on
the internal cluster network and the external lan here). It still
responds to a
query (using the python ganglia client or through the standard front
end web
page) but it stops seeing the client nodes and marks them off-line. It
will indicate
that only one host (itself) is actually up. I have to constantly be
watching the
outputs to see if this has happened and when it does do a:
/etc/init.d/gmond restart
That clears it up until next time.
Any idea what could be causing this? I have been using ganglia to
monitor clusters
for quite some time but this is the first time i have seen the gmond
process needing
to be restarted to regain connection to the data stream running around
inside the cluster.
BTW, I have added a line to the /etc/init.d/gmond script to add a host
route on the
system with the dual network interfaces to point 239.2.11.71 to the
network interface
that faces to the internal network of the cluster.
I do not seem to have this issue with the cluster that has RedHatEL3
installed (same
hardware thought). It is a smaller cluster (64 nodes verses the 128
cluster) though.
--
Steven A. DuChene
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting
language
that extends applications into web and mobile media. Attend the live
webcast
and join the prime developer group breaking into this new coding
territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live
webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
