You'll need a gmond instance running on the collector for each cluster
you want to aggregate for.
So for each cluster, gmond will need the following. Your UDP
send/receive port will obviously need to be unique, since a discreet
gmond process will exist for each "cluster" you're aggregating for that
way. Sorry for the munged info that follows.
/eli
/*SERVER:*/
globals {
mute = "yes"
deaf = "no"
...
cluster {
name = "Opteron_Production-..."
...
udp_send_channel {
host = "10...6"
port = "8701"
...
udp_recv_channel {
port = 8701
family = inet4
And on each node:
/*CLIENT*/
globals {
mute = "no"
deaf = "no"
...
cluster {
name = "Opteron_Production-..."
...
udp_send_channel {
host = "10...6"
port = "8701"
...
udp_recv_channel {
port = 8701
family = inet4
Steven A. DuChene wrote:
If I wanted to try setting up unicast stuff from the individual nodes back
to the head node in an attempt to eliminate my problem with gmond
loosing communication with the rest of the cluster, is there some definate
documentation on what changes have to be done to accomplish that?
-----Original Message-----
From: "Steven A. DuChene" <[EMAIL PROTECTED]>
Sent: Mar 30, 2006 12:52 PM
To: [EMAIL PROTECTED], [email protected], [EMAIL PROTECTED]
Subject: Re: [Ganglia-general] gmond unreliable on one cluster, must be
constantly restarted
The gmetad host is completely seperate from this cluster as I have
one gmetad system outside the cluster that agrigates the gmond stuff from
multiple clusters. The other clusters are just fine. Those other clusters are
running RHEL3 while the problem one is running RHEL4. The gmond on the
problem cluster headnode does not die. It just gets into a state where it is
not reporting the other nodes in the cluster as alive. The way I double check
this outside of the web page display from the gmetad system is I run the
ganglia python command line client directly on the RHEL4 head node exhibiting
the problem. The head node on the problem cluster has to have the gmond
process stopped and restarted to regain the complete cluster in ganglia.
I am using multicasting and the head node does have multiple NICs but none
are teamed. I also see this same issue on other non-compute nodes inside
the cluster but since those are not the ones the outside gmetad system is
pulling stats from it does not seem to affect anything. I.E. when I restart
the gmond process on the headnode, any other systems exhibiting the problem
(as seen from running the ganglia python client on that other system) report
and are shown normally in the ganglia web and python clients.
Because of the 2.6 kernel on the RHEL4 cluster where this problem is occuring
should I try taking out the static host route to 239.2.11.71 to direct the
ganglia
packets to the correct network interface? I have always added this static route
bit based on the information in the ganglia FAQ regarding systems with multiple
network interfaces.
-----Original Message-----
From: Martin Knoblauch <[EMAIL PROTECTED]>
Sent: Mar 29, 2006 11:59 PM
To: "Steven A. DuChene" <[EMAIL PROTECTED]>,
[email protected]
Subject: Re: [Ganglia-general] gmond unreliable on one cluster, must be
constantly restarted
Steven,
do you see anything in the /var/log/messages of the gmetad host? Do
you insert any custom metrics via "gmetric" or other means?
Cheers
Martin
--- "Steven A. DuChene" <[EMAIL PROTECTED]> wrote:
I have been struggling with a gmond process on one cluster here that
after some indeterminate period of time marks everything else in the
cluster as down so instead of having a clustersize (as indicqated
from the
ganglia python command line client) of 135, the clustersize is 1.
I have on the advice of Richard looked at the TN verses TMAX values
in the XML that I get from this host when I telnet to it's port 8649
and the TN values are much bigger than the TMAX values. Richard said
this indicated a problem but I am not sure where to go next with
trying
to diagnose this issue. I have ganglia running on other clusters just
fine
but this one cluster running 3.0.2 on RHEL4u2 seems to be having an
issue.
Any suggestions as to what I can do to continue diagnosing the issue?
-----Original Message-----
From: "Steven A. DuChene" <[EMAIL PROTECTED]>
Sent: Mar 28, 2006 5:48 PM
To: [EMAIL PROTECTED],
[email protected]
Subject: RE: [Ganglia-general] gmond stops recognizing the rest of
the cluster
Yes, I have confirmed that when this condition occurs the TN figures
are MUCH greater
than the TMAX figures. I have double checked the routes and stuff is
still there (i.e.
netstat returns:
Destination Gateway Genmask Flags MSS Window
irtt Iface
239.2.11.71 0.0.0.0 255.255.255.255 UH 0 0
0 eth0
Where the host route for 239.2.11.71 is indeed still associated with
the interface
on the internal cluster network.
Any suggestions then?
-----Original Message-----
From: [EMAIL PROTECTED]
Sent: Mar 22, 2006 4:00 AM
To: [EMAIL PROTECTED],
[email protected]
Subject: RE: [Ganglia-general] gmond stops recognizing the rest of
the cluster
Steven,
if the problem is routing or actual packet loss, then that should
be
reflected
by the XML output of the master gmond - the "down" host will have a
TN
(much) greater
than the TMAX. e.g.:
<HOST NAME="ldndsm030000185.intranet.barcapint.com"
IP="10.68.90.10"
REPORTED="1143022788" TN="145" TMAX="20" DMAX="0"
LOCATION="unspecified"
GMOND_STARTED="1142870107">
There is also a very small chance that what you are seeing is
related to
the "Possible bug in hosts up calculation" thread. This bug causes
an
erroneous
tagging of a data source as "old", which then changes the host up
calculation to be
one based on the wall clock of the gmetad server. Unless all the
clocks
are right,
the host_up calculation is wrong.
You may try this patch:
http://sourceforge.net/mailarchive/message.php?msg_id=15170774
hope springs eternal, anyway. Myself, I only encountered the
problem
fixed here
when I was federating clusters.
There is also a host_up calculation in the PHP web stuff,
ganglia.php,
function host_alive.
You could put debugging in there as well.
kind regards,
Richard
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of
Steven
A. DuChene
Sent: 21 March 2006 23:32
To: [email protected]
Subject: [Ganglia-general] gmond stops recognizing the rest of the
cluster
I have a couple of mixed clusters here with AMD64/Opteron compute
nodes
and Intel EM64T Xeon managment nodes and I am running
ganglia-gmond-3.0.2
Periodically (sometimes a couple or more times a day) I check the
stats
for the clusters and the cluster running RedHatEL-4.0 has a problem
with
the master gmond process (the one running on the management server
with
interfaces on the internal cluster network and the external lan
here).
It still responds to a query (using the python ganglia client or
through
the standard front end web
page) but it stops seeing the client nodes and marks them off-line.
It
will indicate that only one host (itself) is actually up. I have to
constantly be watching the outputs to see if this has happened and
when
it does do a:
/etc/init.d/gmond restart
That clears it up until next time.
Any idea what could be causing this? I have been using ganglia to
monitor clusters for quite some time but this is the first time i
have
seen the gmond process needing to be restarted to regain connection
to
the data stream running around inside the cluster.
BTW, I have added a line to the /etc/init.d/gmond script to add a
host
route on the system with the dual network interfaces to point
239.2.11.71 to the network interface that faces to the internal
network
of the cluster.
I do not seem to have this issue with the cluster that has
RedHatEL3
installed (same hardware thought). It is a smaller cluster (64
nodes
verses the 128 cluster) though.
--
Steven A. DuChene
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting
language
that extends applications into web and mobile media. Attend the live
webcast
and join the prime developer group breaking into this new coding
territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
