Hello everyone, I dont know why I didnt think of this before; all that looking into the code and I ignored the GUI... I found out that if I change the Sorted view at the top from whatever its on to something else, all the nodes show up again. It seems to happen regardless of what choice is made (ascending, descending, hostname), but if I am quick enough when they go "dead" it will not leave a gap on the graphs. Problem is that I have other things to do besides trying to prevent gaps in ganglia. :-D
I guess its not really that big of a deal, but we leave ganglia running on the frontend monitor when we dont have anything going on and its annoying when people ask why ganglia/systems are down after spoting that gaps. Then theres the minor heartattack when a program is deployed to the nodes and ganglia suddenly reports them all as dead :-P Still dont know why it only happens to one group of systems on one monitor though... Anyway, I appreciate any input. Thanks, Chris Stackpole -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of Stackpole, Chris Sent: Thursday, May 18, 2006 12:02 PM To: [email protected] Subject: [Ganglia-general] Ganglia blackouts Good day everyone, I have been noticing a odd problem with ganglia that I hope someone can help with. I was unable to find anything in the mailing list. The short of it is that ganglia will randomly report all the nodes as dead, yet it will still show that it is updating the heartbeats. (link to imageshack for screenshot) http://img231.imageshack.us/img231/2910/snapshot18si.jpg This results in my graphs having gaps. http://img231.imageshack.us/img231/3770/snapshot26sh.jpg While random, this will happen several times a day. Sometimes the gap is so small that it will not show on the day view, sometimes it will be upwards of ten minutes. Occasionally I will get the error message (usually a few days in between that I notice) : Warning: arsort() expects parameters 1 to be array, null given in /var/www/html/ganglia/cluster_view.php on line 95 which is this line in the code switch ($sort) { case "descending": -> arsort($sorted_hosts); break; case "by hostname": ksort($sorted_hosts); break; default: case "ascending": asort($sorted_hosts); break; } Now for the part that really confuses me. I have a second system monitoring other computers; same ganglia version and gmond (3.0.3 ). I have set both of them up so that they will also monitor the other group. Group A has these gaps on its nodes, but will not have them when viewing group B. Group B monitor does not have the gaps at all. If it was one group of systems, I would look at the systems. If one of the monitors was doing it to both groups, I would look at the monitor as the problem. But it is just one monitor with one group and I have no idea what's going on. I have already tried boosting ganglia's priority, and making certain the two groups were on different ports just in case there was a cpu/network flood causing it. To me, if it is still able to update the heartbeat then it "sees" the other systems. Why then report them as dead? (which is why, I assume, the graphs have gaps) I would appreciate any comments and suggestions. Thanks everyone, Chris Stackpole ------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=k&kid0709&bid&3057&dat1642 _______________________________________________ Ganglia-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ganglia-general
