My installation started having an issue yesterday afternoon that I have
yet to explain or remedy. One cluster that I have unicasting, has
started "losing" hosts... the directory entries on disk never get
created for newly deployed hosts, and gmond reports receiving messages
for the host (and outputs metrics) but gmetad does not report an
"updating host" message, and never creates the RRD's even though the
host is up.
The critical problem is that the report graphs for this cluster have
stopped being updated as well, which nix'es my ability to view cluster
load/job level... in addition to not being able to alert on the RRD
values for the individual hosts that are malfunctioning. Those hosts
that are "good" continue to update their metric RRD's properly, their
host reports are populated etc. The bad ones I cannot explain...
The two questions, if anyone has insight:
1) What is causing gmetad to stop acting on the gmond XML input that it
has available? I don't see any error or threshhold it's hitting WRT the
hosts, they just don't create/update the RRD
2) Why does the report stop being populated (the graph is still
generated with past data, but not updated with new... not even the data
from hosts that ARE functioning individually.
I'm continuing on with this, will update with anything else I find awry.
Any suggestions on what to pursue beyond this are welcome... at this
point it looks to me a problem with the magic in gmetad's parsing of the
gmond output, since it is present and up-to-date but not acting on it.
Cheers,
/eli
Here are the details:
server:
ganglia 3.0.2 (x86_64)
6 (six) multicast clusters polled by gmetad
1 (one) unicast cluster, reporting to a 'mute' gmond aggregating on the
same host as gmetad.
clients:
suse9.3 x86_64
ganglia 3.0.2 (x86_64)
Debug logged info (-d2):
Bad host:
Apache error_log for bad host:
ERROR: opening
'/var/lib/ganglia/rrds/Opteron_Production-Desktop_Droid_Cluster/frankenstein.lucasfilm.com/swap_free.rrd':
No such file or directory
gmond:
Processing a Ganglia_message from badhost
gmetad:
server_thread() received request
"/Opteron_Production-Desktop_Droid_Cluster/badhost" from 127.0.0.1
XML:
<HOST NAME="badhost" IP="10.65.34.22" REPORTED="1143682835" TN="4"
TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1143677550">
<METRIC NAME="cpu_num" VAL="2" TYPE="uint16" UNITS="CPUs" TN="488"
TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="disk_total" VAL="71.047" TYPE="double" UNITS="GB"
TN="1688" TMAX="1200" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="disk_free" VAL="57.776" TYPE="double" UNITS="GB" TN="128"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="cpu_speed" VAL="2612" TYPE="uint32" UNITS="MHz" TN="488"
TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="part_max_used" VAL="52.7" TYPE="float" UNITS="" TN="128"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="mem_total" VAL="8147640" TYPE="uint32" UNITS="KB" TN="488"
TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="swap_total" VAL="2104504" TYPE="uint32" UNITS="KB"
TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="boottime" VAL="1143590767" TYPE="uint32" UNITS="s"
TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="machine_type" VAL="x86_64" TYPE="string" UNITS="" TN="488"
TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="os_name" VAL="Linux" TYPE="string" UNITS="" TN="488"
TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="os_release" VAL="2.6.13.4_K8+NUMA+NV" TYPE="string"
UNITS="" TN="488" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="cpu_user" VAL="93.6" TYPE="float" UNITS="%" TN="27"
TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="cpu_system" VAL="0.6" TYPE="float" UNITS="%" TN="27"
TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="load_one" VAL="2.03" TYPE="float" UNITS="" TN="68"
TMAX="70" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="proc_run" VAL="2" TYPE="uint32" UNITS="" TN="8" TMAX="950"
DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="proc_total" VAL="128" TYPE="uint32" UNITS="" TN="8"
TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="mem_free" VAL="1328356" TYPE="uint32" UNITS="KB" TN="8"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="mem_shared" VAL="0" TYPE="uint32" UNITS="KB" TN="8"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="mem_buffers" VAL="199232" TYPE="uint32" UNITS="KB" TN="8"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="mem_cached" VAL="4569200" TYPE="uint32" UNITS="KB" TN="8"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="swap_free" VAL="2101964" TYPE="uint32" UNITS="KB" TN="8"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="gexec" VAL="ON" TYPE="string" UNITS="" TN="188" TMAX="300"
DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="bytes_out" VAL="6066.85" TYPE="float" UNITS="bytes/sec"
TN="8" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="bytes_in" VAL="203006.30" TYPE="float" UNITS="bytes/sec"
TN="8" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="numthreads" VAL="2" TYPE="int8" UNITS="" TN="324"
TMAX="60" DMAX="0" SLOPE="both" SOURCE="gmetric"/>
<METRIC NAME="numjobs" VAL="2" TYPE="int8" UNITS="" TN="324" TMAX="60"
DMAX="0" SLOPE="both" SOURCE="gmetric"/>
</HOST>
Good host:
gmond:
Processing a Ganglia_message from goodhost
gmetad:
Updating host goodhost, metric numjobs
server_thread() received request
"/Opteron_Production-Desktop_Droid_Cluster/goodhost" from 127.0.0.1
XML:
<HOST NAME="goodhost" IP="10.73.16.225" REPORTED="1143682838" TN="1"
TMAX="20" DMAX="0" LOCATION="unspecified" GMOND_STARTED="1143137198">
<METRIC NAME="cpu_num" VAL="2" TYPE="uint16" UNITS="CPUs" TN="838"
TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="disk_total" VAL="71.047" TYPE="double" UNITS="GB"
TN="2039" TMAX="1200" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="disk_free" VAL="46.667" TYPE="double" UNITS="GB" TN="178"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="cpu_speed" VAL="2411" TYPE="uint32" UNITS="MHz" TN="838"
TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="part_max_used" VAL="70.5" TYPE="float" UNITS="" TN="178"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="mem_total" VAL="8147640" TYPE="uint32" UNITS="KB" TN="838"
TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="swap_total" VAL="2104504" TYPE="uint32" UNITS="KB"
TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="boottime" VAL="1142553979" TYPE="uint32" UNITS="s"
TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="machine_type" VAL="x86_64" TYPE="string" UNITS="" TN="838"
TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="os_name" VAL="Linux" TYPE="string" UNITS="" TN="838"
TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="os_release" VAL="2.6.13.4_K8+NUMA+NV" TYPE="string"
UNITS="" TN="838" TMAX="1200" DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="cpu_user" VAL="73.1" TYPE="float" UNITS="%" TN="8"
TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="cpu_system" VAL="3.9" TYPE="float" UNITS="%" TN="8"
TMAX="90" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="load_one" VAL="1.99" TYPE="float" UNITS="" TN="9"
TMAX="70" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="proc_run" VAL="2" TYPE="uint32" UNITS="" TN="149"
TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="proc_total" VAL="156" TYPE="uint32" UNITS="" TN="149"
TMAX="950" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="mem_free" VAL="2359176" TYPE="uint32" UNITS="KB" TN="28"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="mem_shared" VAL="0" TYPE="uint32" UNITS="KB" TN="28"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="mem_buffers" VAL="36384" TYPE="uint32" UNITS="KB" TN="28"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="mem_cached" VAL="4162056" TYPE="uint32" UNITS="KB" TN="28"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="swap_free" VAL="1786428" TYPE="uint32" UNITS="KB" TN="28"
TMAX="180" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="gexec" VAL="ON" TYPE="string" UNITS="" TN="229" TMAX="300"
DMAX="0" SLOPE="zero" SOURCE="gmond"/>
<METRIC NAME="bytes_out" VAL="305162.19" TYPE="float" UNITS="bytes/sec"
TN="28" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="bytes_in" VAL="40802.30" TYPE="float" UNITS="bytes/sec"
TN="28" TMAX="300" DMAX="0" SLOPE="both" SOURCE="gmond"/>
<METRIC NAME="numthreads" VAL="1" TYPE="int8" UNITS="" TN="844"
TMAX="60" DMAX="0" SLOPE="both" SOURCE="gmetric"/>
<METRIC NAME="numjobs" VAL="1" TYPE="int8" UNITS="" TN="844" TMAX="60"
DMAX="0" SLOPE="both" SOURCE="gmetric"/>
</HOST>
