This came to me last night as I was trying to sleep... I wanted to
clarify the example below, the "data_source ... IP:8600" example I'd
keep the same if you are polling remote gmond instances which are
aggregating your unicast cluster statistics. The obvious exception is
when I was suggesting to run multiple gmond collector instances on one
system, when you'd have to use a different tcp_accept_channel for each
cluster and specify it that way in your data_source gmetad config.
Probably obvious, but I never explicitly pointed that out. Sorry for
being pedantic ;)
/eli
Eli Stair wrote:
Steven A. DuChene wrote:
OK, if I have a "collector" gmond running per cluster but I have a
seperate
external agrigator system running gmetad pulling data from all the
seperate
clusters do I setup a udp_send_channel entry to "send" stuff to the
gmetad
agrigator system that sits outside all of the clusters? If so what do
I use for
Nope, gmetad still needs to be configured to pull XML from a
"tcp_accept_channel"... in my setups I always use the same port across
all clusters for the XML port, as it makes doing a variety of things
with different tools much easier. The UDP port I always keep different
whether uni or multicast to avoid tainting things, and to make
recognizing configs and traffic easier.
So the equivalent of:
#GLOBAL, this allows gmetad and any other tools to be coded to use the
same port:
tcp_accept_channel {
port = "8600"
timeout = -1
/* your trusted_hosts assuming ipv4 mask*/
acl{
default="deny"
access {
ip="10...."
mask = 24
action = "allow"
}
}
#PER-CLUSTER, I increment this UDP port by one per cluster instance
#and the IP will be either one host per cluster you wish to collect
#that will be polled by gmetad... ALTERNATELY, this can be the SAME
#IP and have a separate gmond instance on each port running on one host
udp_send_channel {
host = "10..."
port = "8701"
ttl="10"
}
udp_recv_channel {
port = 8701
family = inet4
}
#GMETAD: where each IP is a different host defined for that cluster in
the 'udp_send_channel' above:
data_source "Opteron_Blade1_...." 60 10...1:8600
data_source "Opteron_Blade2_...." 60 10...2:8600
a port number for that external gmetad agrigator system in the
udp_send_channel
entry for each individual internal cluster agrigator gmond.conf? Is it
port
8649, 8651, 8652, or what? In the gmetad.conf file on that external
cluster
gmetad system I have a data_source entry defined for each per cluster
head node collector system.
-----Original Message-----
From: Eli Stair <[EMAIL PROTECTED]>
Sent: Apr 4, 2006 6:35 PM
To: "Steven A. DuChene" <[EMAIL PROTECTED]>
Cc: [EMAIL PROTECTED], [email protected],
[EMAIL PROTECTED]
Subject: Re: [Ganglia-general] gmond unreliable on one cluster, must
be constantly restarted
You'll need a gmond instance running on the collector for each
cluster you want to aggregate for.
So for each cluster, gmond will need the following. Your UDP
send/receive port will obviously need to be unique, since a discreet
gmond process will exist for each "cluster" you're aggregating for
that way. Sorry for the munged info that follows.
/eli
/*SERVER:*/
globals {
mute = "yes"
deaf = "no"
...
cluster {
name = "Opteron_Production-..."
...
udp_send_channel {
host = "10...6"
port = "8701"
...
udp_recv_channel {
port = 8701
family = inet4
And on each node:
/*CLIENT*/
globals {
mute = "no"
deaf = "no"
...
cluster {
name = "Opteron_Production-..."
...
udp_send_channel {
host = "10...6"
port = "8701"
...
udp_recv_channel {
port = 8701
family = inet4
Steven A. DuChene wrote:
If I wanted to try setting up unicast stuff from the individual
nodes back
to the head node in an attempt to eliminate my problem with gmond
loosing communication with the rest of the cluster, is there some
definate
documentation on what changes have to be done to accomplish that?
-----Original Message-----
From: "Steven A. DuChene" <[EMAIL PROTECTED]>
Sent: Mar 30, 2006 12:52 PM
To: [EMAIL PROTECTED], [email protected],
[EMAIL PROTECTED]
Subject: Re: [Ganglia-general] gmond unreliable on one cluster,
must be constantly restarted
The gmetad host is completely seperate from this cluster as I have
one gmetad system outside the cluster that agrigates the gmond
stuff from
multiple clusters. The other clusters are just fine. Those other
clusters are
running RHEL3 while the problem one is running RHEL4. The gmond on the
problem cluster headnode does not die. It just gets into a state
where it is
not reporting the other nodes in the cluster as alive. The way I
double check
this outside of the web page display from the gmetad system is I
run the
ganglia python command line client directly on the RHEL4 head node
exhibiting
the problem. The head node on the problem cluster has to have the
gmond
process stopped and restarted to regain the complete cluster in
ganglia.
I am using multicasting and the head node does have multiple NICs
but none
are teamed. I also see this same issue on other non-compute nodes
inside
the cluster but since those are not the ones the outside gmetad
system is
pulling stats from it does not seem to affect anything. I.E. when I
restart
the gmond process on the headnode, any other systems exhibiting the
problem
(as seen from running the ganglia python client on that other
system) report
and are shown normally in the ganglia web and python clients.
Because of the 2.6 kernel on the RHEL4 cluster where this problem
is occuring
should I try taking out the static host route to 239.2.11.71 to
direct the ganglia
packets to the correct network interface? I have always added this
static route
bit based on the information in the ganglia FAQ regarding systems
with multiple
network interfaces.
-----Original Message-----
From: Martin Knoblauch <[EMAIL PROTECTED]>
Sent: Mar 29, 2006 11:59 PM
To: "Steven A. DuChene" <[EMAIL PROTECTED]>,
[email protected]
Subject: Re: [Ganglia-general] gmond unreliable on one cluster,
must be constantly restarted
Steven,
do you see anything in the /var/log/messages of the gmetad host? Do
you insert any custom metrics via "gmetric" or other means?
Cheers
Martin
--- "Steven A. DuChene" <[EMAIL PROTECTED]> wrote:
I have been struggling with a gmond process on one cluster here that
after some indeterminate period of time marks everything else in the
cluster as down so instead of having a clustersize (as indicqated
from the
ganglia python command line client) of 135, the clustersize is 1.
I have on the advice of Richard looked at the TN verses TMAX values
in the XML that I get from this host when I telnet to it's port 8649
and the TN values are much bigger than the TMAX values. Richard said
this indicated a problem but I am not sure where to go next with
trying
to diagnose this issue. I have ganglia running on other clusters
just
fine
but this one cluster running 3.0.2 on RHEL4u2 seems to be having an
issue.
Any suggestions as to what I can do to continue diagnosing the
issue?
-----Original Message-----
From: "Steven A. DuChene" <[EMAIL PROTECTED]>
Sent: Mar 28, 2006 5:48 PM
To: [EMAIL PROTECTED],
[email protected]
Subject: RE: [Ganglia-general] gmond stops recognizing the rest of
the cluster
Yes, I have confirmed that when this condition occurs the TN
figures
are MUCH greater
than the TMAX figures. I have double checked the routes and
stuff is
still there (i.e.
netstat returns:
Destination Gateway Genmask Flags MSS Window
irtt Iface
239.2.11.71 0.0.0.0 255.255.255.255 UH 0 0
0 eth0
Where the host route for 239.2.11.71 is indeed still associated
with
the interface
on the internal cluster network.
Any suggestions then?
-----Original Message-----
From: [EMAIL PROTECTED]
Sent: Mar 22, 2006 4:00 AM
To: [EMAIL PROTECTED],
[email protected]
Subject: RE: [Ganglia-general] gmond stops recognizing the rest of
the cluster
Steven,
if the problem is routing or actual packet loss, then that should
be
reflected
by the XML output of the master gmond - the "down" host will
have a
TN
(much) greater
than the TMAX. e.g.:
<HOST NAME="ldndsm030000185.intranet.barcapint.com"
IP="10.68.90.10"
REPORTED="1143022788" TN="145" TMAX="20" DMAX="0"
LOCATION="unspecified"
GMOND_STARTED="1142870107">
There is also a very small chance that what you are seeing is
related to
the "Possible bug in hosts up calculation" thread. This bug causes
an
erroneous
tagging of a data source as "old", which then changes the host up
calculation to be
one based on the wall clock of the gmetad server. Unless all the
clocks
are right,
the host_up calculation is wrong.
You may try this patch:
http://sourceforge.net/mailarchive/message.php?msg_id=15170774
hope springs eternal, anyway. Myself, I only encountered the
problem
fixed here
when I was federating clusters.
There is also a host_up calculation in the PHP web stuff,
ganglia.php,
function host_alive.
You could put debugging in there as well.
kind regards,
Richard
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of
Steven
A. DuChene
Sent: 21 March 2006 23:32
To: [email protected]
Subject: [Ganglia-general] gmond stops recognizing the rest of the
cluster
I have a couple of mixed clusters here with AMD64/Opteron compute
nodes
and Intel EM64T Xeon managment nodes and I am running
ganglia-gmond-3.0.2
Periodically (sometimes a couple or more times a day) I check the
stats
for the clusters and the cluster running RedHatEL-4.0 has a
problem
with
the master gmond process (the one running on the management server
with
interfaces on the internal cluster network and the external lan
here).
It still responds to a query (using the python ganglia client or
through
the standard front end web
page) but it stops seeing the client nodes and marks them
off-line.
It
will indicate that only one host (itself) is actually up. I
have to
constantly be watching the outputs to see if this has happened and
when
it does do a:
/etc/init.d/gmond restart
That clears it up until next time.
Any idea what could be causing this? I have been using ganglia to
monitor clusters for quite some time but this is the first time i
have
seen the gmond process needing to be restarted to regain
connection
to
the data stream running around inside the cluster.
BTW, I have added a line to the /etc/init.d/gmond script to add a
host
route on the system with the dual network interfaces to point
239.2.11.71 to the network interface that faces to the internal
network
of the cluster.
I do not seem to have this issue with the cluster that has
RedHatEL3
installed (same hardware thought). It is a smaller cluster (64
nodes
verses the 128 cluster) though.
--
Steven A. DuChene
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting
language
that extends applications into web and mobile media. Attend the live
webcast
and join the prime developer group breaking into this new coding
territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting
language
that extends applications into web and mobile media. Attend the
live webcast
and join the prime developer group breaking into this new coding
territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting
language
that extends applications into web and mobile media. Attend the
live webcast
and join the prime developer group breaking into this new coding
territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting
language
that extends applications into web and mobile media. Attend the live
webcast
and join the prime developer group breaking into this new coding
territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting
language
that extends applications into web and mobile media. Attend the live
webcast
and join the prime developer group breaking into this new coding
territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live
webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Ganglia-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ganglia-general
