Good day everyone, I have been noticing a odd problem with ganglia that I hope someone can help with. I was unable to find anything in the mailing list.
The short of it is that ganglia will randomly report all the nodes as dead, yet it will still show that it is updating the heartbeats. (link to imageshack for screenshot) http://img231.imageshack.us/img231/2910/snapshot18si.jpg This results in my graphs having gaps. http://img231.imageshack.us/img231/3770/snapshot26sh.jpg While random, this will happen several times a day. Sometimes the gap is so small that it will not show on the day view, sometimes it will be upwards of ten minutes. Occasionally I will get the error message (usually a few days in between that I notice) : Warning: arsort() expects parameters 1 to be array, null given in /var/www/html/ganglia/cluster_view.php on line 95 which is this line in the code switch ($sort) { case "descending": -> arsort($sorted_hosts); break; case "by hostname": ksort($sorted_hosts); break; default: case "ascending": asort($sorted_hosts); break; } Now for the part that really confuses me. I have a second system monitoring other computers; same ganglia version and gmond (3.0.3 ). I have set both of them up so that they will also monitor the other group. Group A has these gaps on its nodes, but will not have them when viewing group B. Group B monitor does not have the gaps at all. If it was one group of systems, I would look at the systems. If one of the monitors was doing it to both groups, I would look at the monitor as the problem. But it is just one monitor with one group and I have no idea what's going on. I have already tried boosting ganglia's priority, and making certain the two groups were on different ports just in case there was a cpu/network flood causing it. To me, if it is still able to update the heartbeat then it "sees" the other systems. Why then report them as dead? (which is why, I assume, the graphs have gaps) I would appreciate any comments and suggestions. Thanks everyone, Chris Stackpole
