Martin, et al: I'm getting "...illegal attempt to update using time 1143703242 when last update time is 1143703242 (minimum one second step)..." messages for the improper 'linux.' hosts. I was assuming that gmetad was sorting/indexing the data from those sources by the FQDN which was the same for about 12 new hosts, and thus erroring out when it receives timestamped data from the "same" host that is out of line with the last send period from a nearby node. gmetad is still creating/populating the linux RRD's however.
IMO, since there's no inter-node communication without multicast being enabled (the XML port on the unicast hosts doesn't even output metrics, only the basic info headers, CLUSTER NAME= stanza, and <!ELEMENT * EMPTY>) there (shouldn't) be anything to worry about with other nodes caching the improper data... however, that doesn't mesh with the host I have fixed and still doesn't update. So still trying to work out why gmetad isn't acting on the metrics sent, and logging the equivalent of "Updating host host.fqdn, metric..." for the previously broken, but now proper DNS'd host. And the most unexplainable item so far, that one functional host I noticed is reported the "HOST NAME=" as that of the gmetad server... after a restart it ceased doing that. Thanks for the input, I'm going to restart all the gmond's on the unicast hosts, just to be "sure", and fixing the crufty PTR records as well. Any other thoughts anyone? Thanks, /eli -----Original Message----- From: Martin Knoblauch [mailto:[EMAIL PROTECTED] Sent: Wed 3/29/2006 11:05 PM To: Eli Stair Cc: [email protected] Subject: Re: [Ganglia-general] Re: gmetad not updating RRD's/hosts that are proper in gmond XML Eli, yup. That could definitely cause problems. Do you see anything in the /var/log/messages of the gmetad host? Hmm. You may have to restart *all* gmonds, as well as the gmetad. This is something that I usually do when my ganglia setup was hosed somehow. Definitely the case for multicast clusters. Not really sure about unicast. And yes - this is not optimal. --- Eli Stair <[EMAIL PROTECTED]> wrote: > > The only issue I can find at all with this config is that the new > hosts > have been deployed by someone with two PTR records, both the proper > one > pointing to the A hostname, as well as all having an improper PTR -> > linux."FQDN". > > Is there a potential that gmetad is doing a lookup of both the > forward > and reverse entries for a host before populating it? Unfortunately > removing the invalid entry for a host and restarting gmetad as well > as > the gmond aggregator and the host did not resolve it. > > /eli > > Eli Stair wrote: > > > > My installation started having an issue yesterday afternoon that I > have > > yet to explain or remedy. One cluster that I have unicasting, has > > started "losing" hosts... the directory entries on disk never get > > created for newly deployed hosts, and gmond reports receiving > messages > > for the host (and outputs metrics) but gmetad does not report an > > "updating host" message, and never creates the RRD's even though > the > > host is up. > > > > The critical problem is that the report graphs for this cluster > have > > stopped being updated as well, which nix'es my ability to view > cluster > > load/job level... in addition to not being able to alert on the RRD > > > values for the individual hosts that are malfunctioning. Those > hosts > > that are "good" continue to update their metric RRD's properly, > their > > host reports are populated etc. The bad ones I cannot explain... > > > > The two questions, if anyone has insight: > > > > 1) What is causing gmetad to stop acting on the gmond XML input > that it > > has available? I don't see any error or threshhold it's hitting > WRT the > > hosts, they just don't create/update the RRD > > > > 2) Why does the report stop being populated (the graph is still > > generated with past data, but not updated with new... not even the > data > > from hosts that ARE functioning individually. > > > > I'm continuing on with this, will update with anything else I find > awry. > > Any suggestions on what to pursue beyond this are welcome... at > this > > point it looks to me a problem with the magic in gmetad's parsing > of the > > gmond output, since it is present and up-to-date but not acting on > it. > > > > Cheers, > > > > /eli > > > > > > Here are the details: > > > > server: > > ganglia 3.0.2 (x86_64) > > 6 (six) multicast clusters polled by gmetad > > 1 (one) unicast cluster, reporting to a 'mute' gmond aggregating on > the > > same host as gmetad. > > > > clients: > > suse9.3 x86_64 > > ganglia 3.0.2 (x86_64) > > > > > > Debug logged info (-d2): > > > > Bad host: > > > > Apache error_log for bad host: > > ERROR: opening > > > '/var/lib/ganglia/rrds/Opteron_Production-Desktop_Droid_Cluster/frankenstein.lucasfilm.com/swap_free.rrd': > > > No such file or directory > > > > gmond: > > Processing a Ganglia_message from badhost > > gmetad: > > server_thread() received request > > "/Opteron_Production-Desktop_Droid_Cluster/badhost" from 127.0.0.1 > > > > XML: > > <HOST NAME="badhost" IP="10.65.34.22" REPORTED="1143682835" TN="4" > > TMAX="20" DMAX="0" LOCATION="unspecified" > GMOND_STARTED="1143677550"> > > <METRIC NAME="cpu_num" VAL="2" TYPE="uint16" UNITS="CPUs" TN="488" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="disk_total" VAL="71.047" TYPE="double" UNITS="GB" > > TN="1688" TMAX="1200" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="disk_free" VAL="57.776" TYPE="double" UNITS="GB" > TN="128" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="cpu_speed" VAL="2612" TYPE="uint32" UNITS="MHz" > TN="488" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="part_max_used" VAL="52.7" TYPE="float" UNITS="" > TN="128" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_total" VAL="8147640" TYPE="uint32" UNITS="KB" > TN="488" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="swap_total" VAL="2104504" TYPE="uint32" UNITS="KB" > > TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="boottime" VAL="1143590767" TYPE="uint32" UNITS="s" > > TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="machine_type" VAL="x86_64" TYPE="string" UNITS="" > TN="488" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="os_name" VAL="Linux" TYPE="string" UNITS="" TN="488" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="os_release" VAL="2.6.13.4_K8+NUMA+NV" TYPE="string" > > UNITS="" TN="488" TMAX="1200" DMAX="0" SLOPE="zero" > SOURCE="gmond"/> > > <METRIC NAME="cpu_user" VAL="93.6" TYPE="float" UNITS="%" TN="27" > > TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="cpu_system" VAL="0.6" TYPE="float" UNITS="%" TN="27" > > TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="load_one" VAL="2.03" TYPE="float" UNITS="" TN="68" > > TMAX="70" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="proc_run" VAL="2" TYPE="uint32" UNITS="" TN="8" > TMAX="950" > > DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="proc_total" VAL="128" TYPE="uint32" UNITS="" TN="8" > > TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_free" VAL="1328356" TYPE="uint32" UNITS="KB" > TN="8" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_shared" VAL="0" TYPE="uint32" UNITS="KB" TN="8" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_buffers" VAL="199232" TYPE="uint32" UNITS="KB" > TN="8" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_cached" VAL="4569200" TYPE="uint32" UNITS="KB" > TN="8" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="swap_free" VAL="2101964" TYPE="uint32" UNITS="KB" > TN="8" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="gexec" VAL="ON" TYPE="string" UNITS="" TN="188" > TMAX="300" > > DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="bytes_out" VAL="6066.85" TYPE="float" > UNITS="bytes/sec" > > TN="8" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="bytes_in" VAL="203006.30" TYPE="float" > UNITS="bytes/sec" > > TN="8" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="numthreads" VAL="2" TYPE="int8" UNITS="" TN="324" > > TMAX="60" DMAX="0" SLOPE="both" SOURCE="gmetric"/> > > <METRIC NAME="numjobs" VAL="2" TYPE="int8" UNITS="" TN="324" > TMAX="60" > > DMAX="0" SLOPE="both" SOURCE="gmetric"/> > > </HOST> > > > > > > Good host: > > > > gmond: > > Processing a Ganglia_message from goodhost > > gmetad: > > Updating host goodhost, metric numjobs > > server_thread() received request > > "/Opteron_Production-Desktop_Droid_Cluster/goodhost" from 127.0.0.1 > > XML: > > <HOST NAME="goodhost" IP="10.73.16.225" REPORTED="1143682838" > TN="1" > > TMAX="20" DMAX="0" LOCATION="unspecified" > GMOND_STARTED="1143137198"> > > <METRIC NAME="cpu_num" VAL="2" TYPE="uint16" UNITS="CPUs" TN="838" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="disk_total" VAL="71.047" TYPE="double" UNITS="GB" > > TN="2039" TMAX="1200" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="disk_free" VAL="46.667" TYPE="double" UNITS="GB" > TN="178" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="cpu_speed" VAL="2411" TYPE="uint32" UNITS="MHz" > TN="838" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="part_max_used" VAL="70.5" TYPE="float" UNITS="" > TN="178" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_total" VAL="8147640" TYPE="uint32" UNITS="KB" > TN="838" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="swap_total" VAL="2104504" TYPE="uint32" UNITS="KB" > > TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="boottime" VAL="1142553979" TYPE="uint32" UNITS="s" > > TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="machine_type" VAL="x86_64" TYPE="string" UNITS="" > TN="838" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="os_name" VAL="Linux" TYPE="string" UNITS="" TN="838" > > TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="os_release" VAL="2.6.13.4_K8+NUMA+NV" TYPE="string" > > UNITS="" TN="838" TMAX="1200" DMAX="0" SLOPE="zero" > SOURCE="gmond"/> > > <METRIC NAME="cpu_user" VAL="73.1" TYPE="float" UNITS="%" TN="8" > > TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="cpu_system" VAL="3.9" TYPE="float" UNITS="%" TN="8" > > TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="load_one" VAL="1.99" TYPE="float" UNITS="" TN="9" > > TMAX="70" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="proc_run" VAL="2" TYPE="uint32" UNITS="" TN="149" > > TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="proc_total" VAL="156" TYPE="uint32" UNITS="" TN="149" > > > TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_free" VAL="2359176" TYPE="uint32" UNITS="KB" > TN="28" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_shared" VAL="0" TYPE="uint32" UNITS="KB" TN="28" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_buffers" VAL="36384" TYPE="uint32" UNITS="KB" > TN="28" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="mem_cached" VAL="4162056" TYPE="uint32" UNITS="KB" > TN="28" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="swap_free" VAL="1786428" TYPE="uint32" UNITS="KB" > TN="28" > > TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="gexec" VAL="ON" TYPE="string" UNITS="" TN="229" > TMAX="300" > > DMAX="0" SLOPE="zero" SOURCE="gmond"/> > > <METRIC NAME="bytes_out" VAL="305162.19" TYPE="float" > UNITS="bytes/sec" > > TN="28" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="bytes_in" VAL="40802.30" TYPE="float" > UNITS="bytes/sec" > > TN="28" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/> > > <METRIC NAME="numthreads" VAL="1" TYPE="int8" UNITS="" TN="844" > > TMAX="60" DMAX="0" SLOPE="both" SOURCE="gmetric"/> > > <METRIC NAME="numjobs" VAL="1" TYPE="int8" UNITS="" TN="844" > TMAX="60" > > DMAX="0" SLOPE="both" SOURCE="gmetric"/> > > </HOST> > > > > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Ganglia-general mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/ganglia-general > > ------------------------------------------------------ Martin Knoblauch email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
