OK, if I have a "collector" gmond running per cluster but I have a seperate external agrigator system running gmetad pulling data from all the seperate clusters do I setup a udp_send_channel entry to "send" stuff to the gmetad agrigator system that sits outside all of the clusters? If so what do I use for a port number for that external gmetad agrigator system in the udp_send_channel entry for each individual internal cluster agrigator gmond.conf? Is it port 8649, 8651, 8652, or what? In the gmetad.conf file on that external cluster gmetad system I have a data_source entry defined for each per cluster head node collector system.
-----Original Message----- >From: Eli Stair <[EMAIL PROTECTED]> >Sent: Apr 4, 2006 6:35 PM >To: "Steven A. DuChene" <[EMAIL PROTECTED]> >Cc: [EMAIL PROTECTED], [email protected], [EMAIL PROTECTED] >Subject: Re: [Ganglia-general] gmond unreliable on one cluster, must be >constantly restarted > > >You'll need a gmond instance running on the collector for each cluster >you want to aggregate for. > >So for each cluster, gmond will need the following. Your UDP >send/receive port will obviously need to be unique, since a discreet >gmond process will exist for each "cluster" you're aggregating for that >way. Sorry for the munged info that follows. > >/eli > >/*SERVER:*/ >globals { > mute = "yes" > deaf = "no" >... >cluster { > name = "Opteron_Production-..." >... >udp_send_channel { > host = "10...6" > port = "8701" >... >udp_recv_channel { > port = 8701 > family = inet4 > >And on each node: > >/*CLIENT*/ >globals { > mute = "no" > deaf = "no" >... >cluster { > name = "Opteron_Production-..." >... >udp_send_channel { > host = "10...6" > port = "8701" >... >udp_recv_channel { > port = 8701 > family = inet4 > > > > >Steven A. DuChene wrote: >> If I wanted to try setting up unicast stuff from the individual nodes back >> to the head node in an attempt to eliminate my problem with gmond >> loosing communication with the rest of the cluster, is there some definate >> documentation on what changes have to be done to accomplish that? >> >> -----Original Message----- >>> From: "Steven A. DuChene" <[EMAIL PROTECTED]> >>> Sent: Mar 30, 2006 12:52 PM >>> To: [EMAIL PROTECTED], [email protected], [EMAIL >>> PROTECTED] >>> Subject: Re: [Ganglia-general] gmond unreliable on one cluster, must be >>> constantly restarted >>> >>> The gmetad host is completely seperate from this cluster as I have >>> one gmetad system outside the cluster that agrigates the gmond stuff from >>> multiple clusters. The other clusters are just fine. Those other clusters >>> are >>> running RHEL3 while the problem one is running RHEL4. The gmond on the >>> problem cluster headnode does not die. It just gets into a state where it is >>> not reporting the other nodes in the cluster as alive. The way I double >>> check >>> this outside of the web page display from the gmetad system is I run the >>> ganglia python command line client directly on the RHEL4 head node >>> exhibiting >>> the problem. The head node on the problem cluster has to have the gmond >>> process stopped and restarted to regain the complete cluster in ganglia. >>> >>> I am using multicasting and the head node does have multiple NICs but none >>> are teamed. I also see this same issue on other non-compute nodes inside >>> the cluster but since those are not the ones the outside gmetad system is >>> pulling stats from it does not seem to affect anything. I.E. when I restart >>> the gmond process on the headnode, any other systems exhibiting the problem >>> (as seen from running the ganglia python client on that other system) report >>> and are shown normally in the ganglia web and python clients. >>> >>> Because of the 2.6 kernel on the RHEL4 cluster where this problem is >>> occuring >>> should I try taking out the static host route to 239.2.11.71 to direct the >>> ganglia >>> packets to the correct network interface? I have always added this static >>> route >>> bit based on the information in the ganglia FAQ regarding systems with >>> multiple >>> network interfaces. >>> >>> -----Original Message----- >>>> From: Martin Knoblauch <[EMAIL PROTECTED]> >>>> Sent: Mar 29, 2006 11:59 PM >>>> To: "Steven A. DuChene" <[EMAIL PROTECTED]>, >>>> [email protected] >>>> Subject: Re: [Ganglia-general] gmond unreliable on one cluster, must be >>>> constantly restarted >>>> >>>> Steven, >>>> >>>> do you see anything in the /var/log/messages of the gmetad host? Do >>>> you insert any custom metrics via "gmetric" or other means? >>>> >>>> Cheers >>>> Martin >>>> >>>> --- "Steven A. DuChene" <[EMAIL PROTECTED]> wrote: >>>> >>>>> I have been struggling with a gmond process on one cluster here that >>>>> after some indeterminate period of time marks everything else in the >>>>> cluster as down so instead of having a clustersize (as indicqated >>>>> from the >>>>> ganglia python command line client) of 135, the clustersize is 1. >>>>> I have on the advice of Richard looked at the TN verses TMAX values >>>>> in the XML that I get from this host when I telnet to it's port 8649 >>>>> and the TN values are much bigger than the TMAX values. Richard said >>>>> this indicated a problem but I am not sure where to go next with >>>>> trying >>>>> to diagnose this issue. I have ganglia running on other clusters just >>>>> fine >>>>> but this one cluster running 3.0.2 on RHEL4u2 seems to be having an >>>>> issue. >>>>> >>>>> Any suggestions as to what I can do to continue diagnosing the issue? >>>>> >>>>> -----Original Message----- >>>>>> From: "Steven A. DuChene" <[EMAIL PROTECTED]> >>>>>> Sent: Mar 28, 2006 5:48 PM >>>>>> To: [EMAIL PROTECTED], >>>>> [email protected] >>>>>> Subject: RE: [Ganglia-general] gmond stops recognizing the rest of >>>>> the cluster >>>>>> Yes, I have confirmed that when this condition occurs the TN figures >>>>> are MUCH greater >>>>>> than the TMAX figures. I have double checked the routes and stuff is >>>>> still there (i.e. >>>>>> netstat returns: >>>>>> >>>>>> Destination Gateway Genmask Flags MSS Window >>>>> irtt Iface >>>>>> 239.2.11.71 0.0.0.0 255.255.255.255 UH 0 0 >>>>> 0 eth0 >>>>>> Where the host route for 239.2.11.71 is indeed still associated with >>>>> the interface >>>>>> on the internal cluster network. >>>>>> >>>>>> Any suggestions then? >>>>>> >>>>>> -----Original Message----- >>>>>>> From: [EMAIL PROTECTED] >>>>>>> Sent: Mar 22, 2006 4:00 AM >>>>>>> To: [EMAIL PROTECTED], >>>>> [email protected] >>>>>>> Subject: RE: [Ganglia-general] gmond stops recognizing the rest of >>>>> the cluster >>>>>>> Steven, >>>>>>> >>>>>>> if the problem is routing or actual packet loss, then that should >>>>> be >>>>>>> reflected >>>>>>> by the XML output of the master gmond - the "down" host will have a >>>>> TN >>>>>>> (much) greater >>>>>>> than the TMAX. e.g.: >>>>>>> >>>>>>> <HOST NAME="ldndsm030000185.intranet.barcapint.com" >>>>> IP="10.68.90.10" >>>>>>> REPORTED="1143022788" TN="145" TMAX="20" DMAX="0" >>>>> LOCATION="unspecified" >>>>>>> GMOND_STARTED="1142870107"> >>>>>>> >>>>>>> There is also a very small chance that what you are seeing is >>>>> related to >>>>>>> the "Possible bug in hosts up calculation" thread. This bug causes >>>>> an >>>>>>> erroneous >>>>>>> tagging of a data source as "old", which then changes the host up >>>>>>> calculation to be >>>>>>> one based on the wall clock of the gmetad server. Unless all the >>>>> clocks >>>>>>> are right, >>>>>>> the host_up calculation is wrong. >>>>>>> >>>>>>> >>>>>>> You may try this patch: >>>>>>> http://sourceforge.net/mailarchive/message.php?msg_id=15170774 >>>>>>> >>>>>>> hope springs eternal, anyway. Myself, I only encountered the >>>>> problem >>>>>>> fixed here >>>>>>> when I was federating clusters. >>>>>>> >>>>>>> There is also a host_up calculation in the PHP web stuff, >>>>> ganglia.php, >>>>>>> function host_alive. >>>>>>> You could put debugging in there as well. >>>>>>> >>>>>>> kind regards, >>>>>>> Richard >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: [EMAIL PROTECTED] >>>>>>> [mailto:[EMAIL PROTECTED] On Behalf Of >>>>> Steven >>>>>>> A. DuChene >>>>>>> Sent: 21 March 2006 23:32 >>>>>>> To: [email protected] >>>>>>> Subject: [Ganglia-general] gmond stops recognizing the rest of the >>>>>>> cluster >>>>>>> >>>>>>> >>>>>>> I have a couple of mixed clusters here with AMD64/Opteron compute >>>>> nodes >>>>>>> and Intel EM64T Xeon managment nodes and I am running >>>>>>> ganglia-gmond-3.0.2 >>>>>>> >>>>>>> Periodically (sometimes a couple or more times a day) I check the >>>>> stats >>>>>>> for the clusters and the cluster running RedHatEL-4.0 has a problem >>>>> with >>>>>>> the master gmond process (the one running on the management server >>>>> with >>>>>>> interfaces on the internal cluster network and the external lan >>>>> here). >>>>>>> It still responds to a query (using the python ganglia client or >>>>> through >>>>>>> the standard front end web >>>>>>> page) but it stops seeing the client nodes and marks them off-line. >>>>> It >>>>>>> will indicate that only one host (itself) is actually up. I have to >>>>>>> constantly be watching the outputs to see if this has happened and >>>>> when >>>>>>> it does do a: >>>>>>> >>>>>>> /etc/init.d/gmond restart >>>>>>> >>>>>>> That clears it up until next time. >>>>>>> >>>>>>> Any idea what could be causing this? I have been using ganglia to >>>>>>> monitor clusters for quite some time but this is the first time i >>>>> have >>>>>>> seen the gmond process needing to be restarted to regain connection >>>>> to >>>>>>> the data stream running around inside the cluster. >>>>>>> >>>>>>> BTW, I have added a line to the /etc/init.d/gmond script to add a >>>>> host >>>>>>> route on the system with the dual network interfaces to point >>>>>>> 239.2.11.71 to the network interface that faces to the internal >>>>> network >>>>>>> of the cluster. >>>>>>> >>>>>>> I do not seem to have this issue with the cluster that has >>>>> RedHatEL3 >>>>>>> installed (same hardware thought). It is a smaller cluster (64 >>>>> nodes >>>>>>> verses the 128 cluster) though. >>>>>>> -- >>>>>>> Steven A. DuChene >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------- >>>>> This SF.Net email is sponsored by xPML, a groundbreaking scripting >>>>> language >>>>> that extends applications into web and mobile media. Attend the live >>>>> webcast >>>>> and join the prime developer group breaking into this new coding >>>>> territory! >>>>> >>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >>>>> _______________________________________________ >>>>> Ganglia-general mailing list >>>>> [email protected] >>>>> https://lists.sourceforge.net/lists/listinfo/ganglia-general >>>>> >>>>> >>>> >>>> ------------------------------------------------------ >>>> Martin Knoblauch >>>> email: k n o b i AT knobisoft DOT de >>>> www: http://www.knobisoft.de >>>> >>>> >>>> ------------------------------------------------------- >>>> This SF.Net email is sponsored by xPML, a groundbreaking scripting language >>>> that extends applications into web and mobile media. Attend the live >>>> webcast >>>> and join the prime developer group breaking into this new coding territory! >>>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >>>> _______________________________________________ >>>> Ganglia-general mailing list >>>> [email protected] >>>> https://lists.sourceforge.net/lists/listinfo/ganglia-general >>> >>> >>> >>> >>> ------------------------------------------------------- >>> This SF.Net email is sponsored by xPML, a groundbreaking scripting language >>> that extends applications into web and mobile media. Attend the live webcast >>> and join the prime developer group breaking into this new coding territory! >>> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >>> _______________________________________________ >>> Ganglia-general mailing list >>> [email protected] >>> https://lists.sourceforge.net/lists/listinfo/ganglia-general >> >> >> >> ------------------------------------------------------- >> This SF.Net email is sponsored by xPML, a groundbreaking scripting language >> that extends applications into web and mobile media. Attend the live webcast >> and join the prime developer group breaking into this new coding territory! >> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >> _______________________________________________ >> Ganglia-general mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/ganglia-general >> > > > >------------------------------------------------------- >This SF.Net email is sponsored by xPML, a groundbreaking scripting language >that extends applications into web and mobile media. Attend the live webcast >and join the prime developer group breaking into this new coding territory! >http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 >_______________________________________________ >Ganglia-general mailing list >[email protected] >https://lists.sourceforge.net/lists/listinfo/ganglia-general
