Eli, Martin is most surely right. If you are running an unpatched 3.0.2, let me share with you the many ways it can all go wrong.
gmond generates the hostnames found in the XML stream by reverse DNS lookup only. Its internal structures treat every different IP address it sees as a different host, regardless of what the reverse DNS entry is. So, if you have 1) Incorrect reverse DNS entries such that 2 different hosts reverse map to the same hostname, 2) Or 2 NICs on a host that are not teamed (i.e. 2 different addresses) and the routing allows packets to exit either NIC, hence either source address may be used. 3) Or a DHCP lease renewal that results in a host changing IP addresses. Then what will happen is that the XML stream from the cluster will contain 2 (or more) entries with different IP addrs, but the same name. Even in the DHCP case when only 1 source address is used at a time, gmond will keep the old IP address entry until a timeout, even though it is not being updated. So dups arise again. Now unfortunately, gmetad only uses the HOSTNAME for the RRD files and its own processing. So if there is a duplicated hostname in the XML stream, it will update the RRDs after parsing the first entry, and then again after parsing the second. As these 2 updates to the same RRD files will occur in less than one second, this results in an RRD update error. On unpatched 3.0.2, this then causes THE ENTIRE PROCESSING OF THE CLUSTER TO BE ABANDONED. So some hosts get updated, some not, and the cluster view does not get updated. If you patch this particular issue, you will still get double processing for duped hosts, which can result in them erroneouly being reported as down (for example). phew. long mail. - richard -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Martin Knoblauch Sent: 30 March 2006 08:05 To: Eli Stair Cc: [email protected] Subject: Re: [Ganglia-general] Re: gmetad not updating RRD's/hosts that are proper in gmond XML Eli, yup. That could definitely cause problems. Do you see anything in the /var/log/messages of the gmetad host? Hmm. You may have to restart *all* gmonds, as well as the gmetad. This is something that I usually do when my ganglia setup was hosed somehow. Definitely the case for multicast clusters. Not really sure about unicast. And yes - this is not optimal. --- Eli Stair <[EMAIL PROTECTED]> wrote: > > The only issue I can find at all with this config is that the new > hosts have been deployed by someone with two PTR records, both the > proper one > pointing to the A hostname, as well as all having an improper PTR -> > linux."FQDN". > > Is there a potential that gmetad is doing a lookup of both the forward > and reverse entries for a host before populating it? Unfortunately > removing the invalid entry for a host and restarting gmetad as well > as > the gmond aggregator and the host did not resolve it. > > /eli > > Eli Stair wrote: > > > > My installation started having an issue yesterday afternoon that I > have > > yet to explain or remedy. One cluster that I have unicasting, has > > started "losing" hosts... the directory entries on disk never get > > created for newly deployed hosts, and gmond reports receiving > messages > > for the host (and outputs metrics) but gmetad does not report an > > "updating host" message, and never creates the RRD's even though > the > > host is up. > > > > The critical problem is that the report graphs for this cluster > have > > stopped being updated as well, which nix'es my ability to view > cluster > > load/job level... in addition to not being able to alert on the RRD > > > values for the individual hosts that are malfunctioning. Those > hosts > > that are "good" continue to update their metric RRD's properly, > their > > host reports are populated etc. The bad ones I cannot explain... > > > > The two questions, if anyone has insight: > > > > 1) What is causing gmetad to stop acting on the gmond XML input > that it > > has available? I don't see any error or threshhold it's hitting > WRT the > > hosts, they just don't create/update the RRD > > > > 2) Why does the report stop being populated (the graph is still > > generated with past data, but not updated with new... not even the > data > > from hosts that ARE functioning individually. > > > > I'm continuing on with this, will update with anything else I find > awry. > > Any suggestions on what to pursue beyond this are welcome... at > this > > point it looks to me a problem with the magic in gmetad's parsing > of the > > gmond output, since it is present and up-to-date but not acting on > it. > > > > Cheers, > > > > /eli > > > > > > Here are the details: > > > > server: > > ganglia 3.0.2 (x86_64) > > 6 (six) multicast clusters polled by gmetad > > 1 (one) unicast cluster, reporting to a 'mute' gmond aggregating on > the > > same host as gmetad. > > > > clients: > > suse9.3 x86_64 > > ganglia 3.0.2 (x86_64) > > > > > > Debug logged info (-d2): > > > > Bad host: > > > > Apache error_log for bad host: > > ERROR: opening > > > '/var/lib/ganglia/rrds/Opteron_Production-Desktop_Droid_Cluster/frankens tein.lucasfilm.com/swap_free.rrd': > > > No such file or directory > > > > gmond: > > Processing a Ganglia_message from badhost > > gmetad: > > server_thread() received request > > "/Opteron_Production-Desktop_Droid_Cluster/badhost" from 127.0.0.1 > > > > XML: > > <HOST NAME="badhost" IP="10.65.34.22" REPORTED="1143682835" TN="4" > > TMAX="20" DMAX="0" LOCATION="unspecified" > GMOND_STARTED="1143677550"> > > <METRIC NAME="cpu_num" VAL="2" TYPE="uint16" UNITS="CPUs" TN="488" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="disk_total" VAL="71.047" TYPE="double" UNITS="GB" > > > > TN="1688" TMAX="1200" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="disk_free" VAL="57.776" TYPE="double" UNITS="GB" > TN="128" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="cpu_speed" VAL="2612" TYPE="uint32" UNITS="MHz" > TN="488" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="part_max_used" VAL="52.7" TYPE="float" UNITS="" > TN="128" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_total" VAL="8147640" TYPE="uint32" UNITS="KB" > TN="488" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="swap_total" VAL="2104504" TYPE="uint32" UNITS="KB" > > TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="boottime" VAL="1143590767" TYPE="uint32" UNITS="s" > > TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="machine_type" VAL="x86_64" TYPE="string" UNITS="" > TN="488" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="os_name" VAL="Linux" TYPE="string" UNITS="" TN="488" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="os_release" VAL="2.6.13.4_K8+NUMA+NV" TYPE="string" > > > > UNITS="" TN="488" TMAX="1200" DMAX="0" SLOPE="zero" > SOURCE="gmond"/> > > <METRIC NAME="cpu_user" VAL="93.6" TYPE="float" UNITS="%" TN="27" > > TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="cpu_system" VAL="0.6" TYPE="float" UNITS="%" TN="27" > > TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="load_one" VAL="2.03" TYPE="float" UNITS="" TN="68" > > TMAX="70" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="proc_run" VAL="2" TYPE="uint32" UNITS="" TN="8" > TMAX="950" > > DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="proc_total" VAL="128" TYPE="uint32" UNITS="" TN="8" > > TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_free" VAL="1328356" TYPE="uint32" UNITS="KB" > TN="8" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_shared" VAL="0" TYPE="uint32" UNITS="KB" TN="8" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_buffers" VAL="199232" TYPE="uint32" UNITS="KB" > TN="8" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_cached" VAL="4569200" TYPE="uint32" UNITS="KB" > TN="8" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="swap_free" VAL="2101964" TYPE="uint32" UNITS="KB" > TN="8" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="gexec" VAL="ON" TYPE="string" UNITS="" TN="188" > TMAX="300" > > DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="bytes_out" VAL="6066.85" TYPE="float" > UNITS="bytes/sec" > > TN="8" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC > > NAME="bytes_in" VAL="203006.30" TYPE="float" > UNITS="bytes/sec" > > TN="8" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC > > NAME="numthreads" VAL="2" TYPE="int8" UNITS="" TN="324" TMAX="60" > > DMAX="0" SLOPE="both" SOURCE="gmetric"/> <METRIC NAME="numjobs" > > VAL="2" > > TYPE="int8" UNITS="" TN="324" > TMAX="60" > > DMAX="0" SLOPE="both" SOURCE="gmetric"/> > > </HOST> > > > > > > Good host: > > > > gmond: > > Processing a Ganglia_message from goodhost > > gmetad: > > Updating host goodhost, metric numjobs > > server_thread() received request > > "/Opteron_Production-Desktop_Droid_Cluster/goodhost" from 127.0.0.1 > > XML: > > <HOST NAME="goodhost" IP="10.73.16.225" REPORTED="1143682838" > TN="1" > > TMAX="20" DMAX="0" LOCATION="unspecified" > GMOND_STARTED="1143137198"> > > <METRIC NAME="cpu_num" VAL="2" TYPE="uint16" UNITS="CPUs" TN="838" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="disk_total" VAL="71.047" TYPE="double" UNITS="GB" > > > > TN="2039" TMAX="1200" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="disk_free" VAL="46.667" TYPE="double" UNITS="GB" > TN="178" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="cpu_speed" VAL="2411" TYPE="uint32" UNITS="MHz" > TN="838" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="part_max_used" VAL="70.5" TYPE="float" UNITS="" > TN="178" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_total" VAL="8147640" TYPE="uint32" UNITS="KB" > TN="838" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="swap_total" VAL="2104504" TYPE="uint32" UNITS="KB" > > TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="boottime" VAL="1142553979" TYPE="uint32" UNITS="s" > > TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="machine_type" VAL="x86_64" TYPE="string" UNITS="" > TN="838" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="os_name" VAL="Linux" TYPE="string" UNITS="" TN="838" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="os_release" VAL="2.6.13.4_K8+NUMA+NV" TYPE="string" > > > > UNITS="" TN="838" TMAX="1200" DMAX="0" SLOPE="zero" > SOURCE="gmond"/> > > <METRIC NAME="cpu_user" VAL="73.1" TYPE="float" UNITS="%" TN="8" > > TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="cpu_system" VAL="3.9" TYPE="float" UNITS="%" TN="8" > > TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="load_one" VAL="1.99" TYPE="float" UNITS="" TN="9" > > TMAX="70" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="proc_run" VAL="2" TYPE="uint32" UNITS="" TN="149" > > TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="proc_total" VAL="156" TYPE="uint32" UNITS="" TN="149" > > > TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_free" VAL="2359176" TYPE="uint32" UNITS="KB" > TN="28" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_shared" VAL="0" TYPE="uint32" UNITS="KB" TN="28" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_buffers" VAL="36384" TYPE="uint32" UNITS="KB" > TN="28" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_cached" VAL="4162056" TYPE="uint32" UNITS="KB" > TN="28" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="swap_free" VAL="1786428" TYPE="uint32" UNITS="KB" > TN="28" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="gexec" VAL="ON" TYPE="string" UNITS="" TN="229" > TMAX="300" > > DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="bytes_out" VAL="305162.19" TYPE="float" > UNITS="bytes/sec" > > TN="28" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC > > NAME="bytes_in" VAL="40802.30" TYPE="float" > UNITS="bytes/sec" > > TN="28" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> <METRIC > > NAME="numthreads" VAL="1" TYPE="int8" UNITS="" TN="844" TMAX="60" > > DMAX="0" SLOPE="both" SOURCE="gmetric"/> <METRIC NAME="numjobs" > > VAL="1" > > TYPE="int8" UNITS="" TN="844" > TMAX="60" > > DMAX="0" SLOPE="both" SOURCE="gmetric"/> > > </HOST> > > > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language that extends applications into web and mobile media. Attend > the live webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Ganglia-general mailing list [email protected] > https://lists.sourceforge.net/lists/listinfo/ganglia-general > > ------------------------------------------------------ Martin Knoblauch email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de ------------------------------------------------------- This SF.Net email is sponsored by xPML, a groundbreaking scripting language that extends applications into web and mobile media. Attend the live webcast and join the prime developer group breaking into this new coding territory! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 _______________________________________________ Ganglia-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ganglia-general ------------------------------------------------------------------------ For more information about Barclays Capital, please visit our web site at http://www.barcap.com. Internet communications are not secure and therefore the Barclays Group does not accept legal responsibility for the contents of this message. Although the Barclays Group operates anti-virus programmes, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Barclays Group. Replies to this email may be monitored by the Barclays Group for operational or business reasons. ------------------------------------------------------------------------
