Yes, all nodes are still broadcasting, but one by one become deaf until all the cluster is ignoring the broadcasted data.
I was already using Monit (http://mmonit.com/monit/) to solve the problem of gmond being unable to respond to a request for the state of the cluster on the XML TCP port. So, I ended up writing a shell script that connects to the local gmond, and checks if the local HOST entry has expired. (TN > TMAX). If the entry has expired then the script restart gmond. The shell script is called by Monit on a periodic basis. Below is a segment of the shell script and the monitrc file - the configuration file for Monit- similar to what I ended up using. ********** test-gmond.sh ************* exec 3<>/dev/tcp/10.10.111.11/8649 cat <&3 > /opt/ganglia/logs/gmond-sample.xml HOSTNAME=`hostname` TN=`cat /opt/ganglia/logs/gmond-sample.xml | grep "<HOST " | grep ${HOSTNAME} | sed 's/.*TN=\"\([0-9]*\)\".*/\1/'` TMAX=`cat /opt/ganglia/logs/gmond-sample.xml | grep "<HOST " | grep ${HOSTNAME} | sed 's/.*TMAX=\"\([0-9]*\)\".*/\1/'` RESTART=0 if [ "${TN}" = "" ] ; then RESTART=1 elif [ "${TMAX}" = "" ] ; then RESTART=1 elif [ ${TN} -gt ${TMAX} ] ; then RESTART=1 fi if [ ${RESTART} -eq 1 ] ; then # echo "Restart required." rm /opt/ganglia/logs/gmond-sample.xml # echo ${GMOND_INIT_SCRIPT} restart /etc/init.d/gmond restart fi ************* monitrc **************** # check every 60 seconds (1 minutes) with an initial delay of 60 seconds set daemon 60 with start delay 60 # This statement is needed if you configure monit to be run by init # set init set pidfile /opt/ganglia/logs/monit.pid set idfile /opt/ganglia/logs/monit.id set statefile /opt/ganglia/logs/.monit.state # Configure the log output set logfile /opt/ganglia/logs/monit.log # Configure a pointer to the SMTP mail server set mailserver smtp.example.com # PORT portnumber # USERNAME username # PASSWORD password # set mail-format { from: [email protected] } # set eventqueue basedir /opt/ganglia/monitqueue slots 100 # Set a recipient of alerts, only send a new alert on the same event # after 6 cycles (since our cycle is set to 1 minutes, then this means that # a reminder is sent every 6 minutes) set alert [email protected] with reminder on 6 cycles # Enable the HTTP web monitor without credentials set httpd port 2180 and use the address 10.100.11.111 #------------------------------------------------------------------------------ # Rule 1 : Restart Gmond if a connection can't be established or if the # reply does not start with the XML prolog # (Do we still need this rule after adding rule 2 ?) #------------------------------------------------------------------------------ check host yourHost with address 10.100.11.111 if failed host 10.100.11.111 port 8649 expect "^<\?xml.*" then exec "/etc/init.d/gmond restart" #------------------------------------------------------------------------------ # Rule 2 : Run test-gmond.sh if the file gmond-sample.xml does not exist or if # it has a timestamp older than 300 seconds (5 minutes). #------------------------------------------------------------------------------ check file gmond-sample.xml with path /opt/ganglia/logs/gmond-sample.xml alert [email protected] only on { nonexist } start program = "/opt/ganglia/bin/test-gmond.sh 10.100.11.111 8649 /etc/init.d/gmond" if timestamp > 300 second then exec "/opt/ganglia/bin/test-gmond.sh 10.100.11.111 8649 /etc/init.d/gmond" -----Original Message----- From: River Tarnell [mailto:[email protected]] Sent: Tuesday, November 17, 2009 7:18 AM To: Rick Cobb Cc: Jorge Medina; [email protected] Subject: Re: [Ganglia-general] gmond 3.1.2 becomes deaf in Solaris SPARC -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Rick Cobb: > We had the same problem with gmond 3.0.4 on Solaris 10 / x86. As far > as we were able to debug, it's a bug in Solaris itself, and > particularly with the interaction between IGMPv3 support in the kernel > and switches that only do IGMPv2. The only workarounds we were able to > use were unicast, or restarting gmond quite often on the machines we > had gmetad talking to. if this were the case, would a visible symptom be that 'snoop' running on the gmond hosts would not see any traffic from other systems? in our case, we can see the gmond traffic, the running gmond just ignores it for some reason. - river. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAksClF4ACgkQIXd7fCuc5vIuAgCghTiqNd014JGwMqLth9GgpLjF xWwAnA5dWN8FfXQsEzUvd5KVG1krvt+u =DFpu -----END PGP SIGNATURE----- ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Ganglia-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ganglia-general
