This is the classic behaviour that comes from a "trunucated" XML stream. There is now a full patch for this in the CVS repository. But if you suffer from this, you should get a /var/log/messages entry like:
Mar 30 09:56:37 ldndsr0163 /apps/ganglia/sbin/gmetad[15336]: Process XML (LDN FIP QA Scenarios PDN): XML_ParseBuffer() error at line 374: no element found do you? This error causes the entire processing for the current opll to be abandoned - the remaining hosts are not updated, nor the cluster summary. - Richard -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Eli Stair Sent: 30 March 2006 02:49 To: [email protected]; [EMAIL PROTECTED] Subject: [Ganglia-general] gmetad not updating RRD's/hosts that are proper in gmond XML My installation started having an issue yesterday afternoon that I have yet to explain or remedy. One cluster that I have unicasting, has started "losing" hosts... the directory entries on disk never get created for newly deployed hosts, and gmond reports receiving messages for the host (and outputs metrics) but gmetad does not report an "updating host" message, and never creates the RRD's even though the host is up. The critical problem is that the report graphs for this cluster have stopped being updated as well, which nix'es my ability to view cluster load/job level... in addition to not being able to alert on the RRD values for the individual hosts that are malfunctioning. Those hosts that are "good" continue to update their metric RRD's properly, their host reports are populated etc. The bad ones I cannot explain... The two questions, if anyone has insight: 1) What is causing gmetad to stop acting on the gmond XML input that it has available? I don't see any error or threshhold it's hitting WRT the hosts, they just don't create/update the RRD 2) Why does the report stop being populated (the graph is still generated with past data, but not updated with new... not even the data from hosts that ARE functioning individually. I'm continuing on with this, will update with anything else I find awry. Any suggestions on what to pursue beyond this are welcome... at this point it looks to me a problem with the magic in gmetad's parsing of the gmond output, since it is present and up-to-date but not acting on it. Cheers, /eli Here are the details: server: ganglia 3.0.2 (x86_64) 6 (six) multicast clusters polled by gmetad 1 (one) unicast cluster, reporting to a 'mute' gmond aggregating on the same host as gmetad. clients: suse9.3 x86_64 ganglia 3.0.2 (x86_64) Debug logged info (-d2): Bad host: Apache error_log for bad host: ERROR: opening '/var/lib/ganglia/rrds/Opteron_Production-Desktop_Droid_Cluster/frankens tein.lucasfilm.com/swap_free.rrd': No such file or directory gmond: Processing a Ganglia_message from badhost gmetad: server_thread() received request "/Opteron_Production-Desktop_Droid_Cluster/badhost" from 127.0.0.1 XML: <HOST NAME="badhost" IP="10.65.34.22" REPORTED="1143682835" TN="4" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1143677550"> <METRIC NAME="cpu_num" VAL="2" TYPE="uint16" UNITS="CPUs" TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="disk_total" VAL="71.047" TYPE="double" UNITS="GB" TN="1688" TMAX="1200" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="disk_free" VAL="57.776" TYPE="double" UNITS="GB" TN="128" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="cpu_speed" VAL="2612" TYPE="uint32" UNITS="MHz" TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="part_max_used" VAL="52.7" TYPE="float" UNITS="" TN="128" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="mem_total" VAL="8147640" TYPE="uint32" UNITS="KB" TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="swap_total" VAL="2104504" TYPE="uint32" UNITS="KB" TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="boottime" VAL="1143590767" TYPE="uint32" UNITS="s" TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="machine_type" VAL="x86_64" TYPE="string" UNITS="" TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="os_name" VAL="Linux" TYPE="string" UNITS="" TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="os_release" VAL="2.6.13.4_K8+NUMA+NV" TYPE="string" UNITS="" TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="cpu_user" VAL="93.6" TYPE="float" UNITS="%" TN="27" TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="cpu_system" VAL="0.6" TYPE="float" UNITS="%" TN="27" TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="load_one" VAL="2.03" TYPE="float" UNITS="" TN="68" TMAX="70" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="proc_run" VAL="2" TYPE="uint32" UNITS="" TN="8" TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="proc_total" VAL="128" TYPE="uint32" UNITS="" TN="8" TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="mem_free" VAL="1328356" TYPE="uint32" UNITS="KB" TN="8" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="mem_shared" VAL="0" TYPE="uint32" UNITS="KB" TN="8" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="mem_buffers" VAL="199232" TYPE="uint32" UNITS="KB" TN="8" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="mem_cached" VAL="4569200" TYPE="uint32" UNITS="KB" TN="8" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="swap_free" VAL="2101964" TYPE="uint32" UNITS="KB" TN="8" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="gexec" VAL="ON" TYPE="string" UNITS="" TN="188" TMAX="300" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="bytes_out" VAL="6066.85" TYPE="float" UNITS="bytes/sec" TN="8" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="bytes_in" VAL="203006.30" TYPE="float" UNITS="bytes/sec" TN="8" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="numthreads" VAL="2" TYPE="int8" UNITS="" TN="324" TMAX="60" DMAX="0" SLOPE="both" SOURCE="gmetric"/> <METRIC NAME="numjobs" VAL="2" TYPE="int8" UNITS="" TN="324" TMAX="60" DMAX="0" SLOPE="both" SOURCE="gmetric"/> </HOST> Good host: gmond: Processing a Ganglia_message from goodhost gmetad: Updating host goodhost, metric numjobs server_thread() received request "/Opteron_Production-Desktop_Droid_Cluster/goodhost" from 127.0.0.1 XML: <HOST NAME="goodhost" IP="10.73.16.225" REPORTED="1143682838" TN="1" TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1143137198"> <METRIC NAME="cpu_num" VAL="2" TYPE="uint16" UNITS="CPUs" TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="disk_total" VAL="71.047" TYPE="double" UNITS="GB" TN="2039" TMAX="1200" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="disk_free" VAL="46.667" TYPE="double" UNITS="GB" TN="178" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="cpu_speed" VAL="2411" TYPE="uint32" UNITS="MHz" TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="part_max_used" VAL="70.5" TYPE="float" UNITS="" TN="178" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="mem_total" VAL="8147640" TYPE="uint32" UNITS="KB" TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="swap_total" VAL="2104504" TYPE="uint32" UNITS="KB" TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="boottime" VAL="1142553979" TYPE="uint32" UNITS="s" TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="machine_type" VAL="x86_64" TYPE="string" UNITS="" TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="os_name" VAL="Linux" TYPE="string" UNITS="" TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="os_release" VAL="2.6.13.4_K8+NUMA+NV" TYPE="string" UNITS="" TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="cpu_user" VAL="73.1" TYPE="float" UNITS="%" TN="8" TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="cpu_system" VAL="3.9" TYPE="float" UNITS="%" TN="8" TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="load_one" VAL="1.99" TYPE="float" UNITS="" TN="9" TMAX="70" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="proc_run" VAL="2" TYPE="uint32" UNITS="" TN="149" TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="proc_total" VAL="156" TYPE="uint32" UNITS="" TN="149" TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="mem_free" VAL="2359176" TYPE="uint32" UNITS="KB" TN="28" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="mem_shared" VAL="0" TYPE="uint32" UNITS="KB" TN="28" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="mem_buffers" VAL="36384" TYPE="uint32" UNITS="KB" TN="28" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="mem_cached" VAL="4162056" TYPE="uint32" UNITS="KB" TN="28" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="swap_free" VAL="1786428" TYPE="uint32" UNITS="KB" TN="28" TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="gexec" VAL="ON" TYPE="string" UNITS="" TN="229" TMAX="300" DMAX="0" SLOPE="zero" SOURCE="gmond"/> <METRIC NAME="bytes_out" VAL="305162.19" TYPE="float" UNITS="bytes/sec" TN="28" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="bytes_in" VAL="40802.30" TYPE="float" UNITS="bytes/sec" TN="28" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC NAME="numthreads" VAL="1" TYPE="int8" UNITS="" TN="844" TMAX="60" DMAX="0" SLOPE="both" SOURCE="gmetric"/> <METRIC NAME="numjobs" VAL="1" TYPE="int8" UNITS="" TN="844" TMAX="60" DMAX="0" SLOPE="both" SOURCE="gmetric"/> </HOST> ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Ganglia-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ganglia-general ------------------------------------------------------------------------ For more information about Barclays Capital, please visit our web site at http://www.barcap.com. Internet communications are not secure and therefore the Barclays Group does not accept legal responsibility for the contents of this message. Although the Barclays Group operates anti-virus programmes, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Barclays Group. Replies to this email may be monitored by the Barclays Group for operational or business reasons. ------------------------------------------------------------------------
