Chris, I saw this behavior when the disk I was using was too heavily used. gmond just dropped stuff (i.e. it never got to write it into the RRDs), and the effect was hosts that were marked down and blanks in the graphs.
The host I was using for monitoring 100ish servers was also doing very disk-heavy DB crunching. -ben On Thu, May 18, 2006 at 12:02:14PM -0500, Stackpole, Chris wrote: > Good day everyone, > > I have been noticing a odd problem with ganglia that I hope someone can help > with. I was unable to find anything in the mailing list. > > The short of it is that ganglia will randomly report all the nodes as dead, > yet it will still show that it is updating the heartbeats. > (link to imageshack for screenshot) > http://img231.imageshack.us/img231/2910/snapshot18si.jpg > > This results in my graphs having gaps. > http://img231.imageshack.us/img231/3770/snapshot26sh.jpg > > While random, this will happen several times a day. Sometimes the gap is so > small that it will not show on the day view, sometimes it will be upwards of > ten minutes. Occasionally I will get the error message (usually a few days in > between that I notice) : > > Warning: arsort() expects parameters 1 to be array, null given in > /var/www/html/ganglia/cluster_view.php on line 95 > > which is this line in the code > switch ($sort) > { > case "descending": > -> arsort($sorted_hosts); > break; > case "by hostname": > ksort($sorted_hosts); > break; > default: > case "ascending": > asort($sorted_hosts); > break; > } > > Now for the part that really confuses me. I have a second system monitoring > other computers; same ganglia version and gmond (3.0.3 ). I have set both of > them up so that they will also monitor the other group. Group A has these > gaps on its nodes, but will not have them when viewing group B. Group B > monitor does not have the gaps at all. > > If it was one group of systems, I would look at the systems. If one of the > monitors was doing it to both groups, I would look at the monitor as the > problem. But it is just one monitor with one group and I have no idea what's > going on. > > I have already tried boosting ganglia's priority, and making certain the two > groups were on different ports just in case there was a cpu/network flood > causing it. To me, if it is still able to update the heartbeat then it "sees" > the other systems. Why then report them as dead? (which is why, I assume, the > graphs have gaps) > > I would appreciate any comments and suggestions. > Thanks everyone, > Chris Stackpole > > > ------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid0709&bid&3057&dat1642 > _______________________________________________ > Ganglia-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/ganglia-general -- Ben Hartshorne email: [EMAIL PROTECTED] http://ben.hartshorne.net
signature.asc
Description: Digital signature
