I have a couple of mixed clusters here with AMD64/Opteron compute nodes and Intel EM64T Xeon managment nodes and I am running ganglia-gmond-3.0.2
Periodically (sometimes a couple or more times a day) I check the stats for the clusters and the cluster running RedHatEL-4.0 has a problem with the master gmond process (the one running on the management server with interfaces on the internal cluster network and the external lan here). It still responds to a query (using the python ganglia client or through the standard front end web page) but it stops seeing the client nodes and marks them off-line. It will indicate that only one host (itself) is actually up. I have to constantly be watching the outputs to see if this has happened and when it does do a: /etc/init.d/gmond restart That clears it up until next time. Any idea what could be causing this? I have been using ganglia to monitor clusters for quite some time but this is the first time i have seen the gmond process needing to be restarted to regain connection to the data stream running around inside the cluster. BTW, I have added a line to the /etc/init.d/gmond script to add a host route on the system with the dual network interfaces to point 239.2.11.71 to the network interface that faces to the internal network of the cluster. I do not seem to have this issue with the cluster that has RedHatEL3 installed (same hardware thought). It is a smaller cluster (64 nodes verses the 128 cluster) though. -- Steven A. DuChene
