Hey Richard, Sure enough, it's the poll rate. Thanks! I updated the default poll rate of 15 seconds to 60 and reran my tests. The RRD last mod time is now changing every minute while updates are being submitted to gmond. However, I don't understand why this should be done. I have a script (attached as create_matrix.sh.txt) that submits spoofed metrics at a known rate. It's ugly but it gets the job done. The first time it goes through it goes at full steam. Subsequent times are gated by a calculated sleep interval. If I run './create_matrix.sh 1000 1 5' it will submit 1 metric for 1000 hosts at a rate of 5 metric updates per second. All up, it takes 200 seconds to run through loops. If I look at the modification time of one of the resulting RRD files, I see that it's updated roughly every 60 seconds, even if I'm only supplying an updated metric every 200. Any idea why? If the metric hasn't changed (the TN value observed in gmond XML output isn't reset to zero) why would gmetad be touching the RRD? I've attached the rrdtool info output for one of my RRDs, the config files for gmond and gmetad, and as mentioned, my script for submitting updates. If you run it with the above parameters and then do 'ls --full-time /var/lib/ganglia/rrds/unspecified/host.0/metric.0.rrd' every few seconds you'll eventually see the touches occurring at the poll rate. The attached gmetad.conf still has the default 15 set. Dave. _____
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Wednesday, August 30, 2006 2:43 AM To: Dr. Dave Blunt; [email protected] Subject: RE: [Ganglia-general] Gmetad RRD update question Dave, I tried this, and for me last_update gets changed at the same rate as my poll rate, so I don't see what you see. What is your poll rate of the cluster in gmetad.conf? Perhaps you could mail me the RRA lines in gmetad.conf and the full output of rrdtool info. I'm sure you aleady know this, but when the rrd file is created, it takes the step size from the poll rate, but if you subsequently change your poll rate in gmetad.conf, ganglia does not try to change the step size in the rrd file. kind regards, Richard -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Dr. Dave Blunt Sent: 29 August 2006 19:15 To: ganglia-general Subject: [Ganglia-general] Gmetad RRD update question I'm trying to stress test gmetad by submitting large numbers of metric updates to a single gmond that gmetad is using as a data_source. When I submit an update to a single metric on a single host and then look at 'rrdtool info' output for the /var/lib/ganglia/rrds/unspecified/hostX/metricY.rrd file, I see a total of 5 changes to that last_update timestamp over a space of 60 seconds or so. When I apply another update to the metric (via gmetric) to gmond the same thing happens. Any ideas as to why the RRD has to be touched five times for every update? Again, this isn't a summary RRD, it's just the RRD recording a single metric on a single host within a cluster. Dave. Dr. Dave Blunt Regional Operations Manager GROUNDWORK Open Source, Inc. 139 Townsend Street, Suite 100 San Francisco, CA 94107-1946 415.992.4500 (office) 206.282.2867 (direct) 650.619.5256 (mobile) 206.374.2892 (fax) [EMAIL PROTECTED] www.groundworkopensource.com ------------------------------------------------------------------------ For more information about Barclays Capital, please visit our web site at http://www.barcap.com. Internet communications are not secure and therefore the Barclays Group does not accept legal responsibility for the contents of this message. Although the Barclays Group operates anti-virus programmes, it does not accept responsibility for any damage whatsoever that is caused by viruses being passed. Any views or opinions presented are solely those of the author and do not necessarily represent those of the Barclays Group. Replies to this email may be monitored by the Barclays Group for operational or business reasons. ------------------------------------------------------------------------
filename = "/var/lib/ganglia/rrds/unspecified/host.0/metric.0.rrd" rrd_version = "0003" step = 15 last_update = 1156953216 ds[sum].type = "GAUGE" ds[sum].minimal_heartbeat = 120 ds[sum].min = NaN ds[sum].max = NaN ds[sum].last_ds = "UNKN" ds[sum].value = NaN ds[sum].unknown_sec = 6 rra[0].cf = "AVERAGE" rra[0].rows = 240 rra[0].pdp_per_row = 1 rra[0].xff = 5.0000000000e-01 rra[0].cdp_prep[0].value = NaN rra[0].cdp_prep[0].unknown_datapoints = 0 rra[1].cf = "AVERAGE" rra[1].rows = 240 rra[1].pdp_per_row = 24 rra[1].xff = 5.0000000000e-01 rra[1].cdp_prep[0].value = 4.6278120560e+09 rra[1].cdp_prep[0].unknown_datapoints = 18 rra[2].cf = "AVERAGE" rra[2].rows = 240 rra[2].pdp_per_row = 168 rra[2].xff = 5.0000000000e-01 rra[2].cdp_prep[0].value = 4.6278120560e+09 rra[2].cdp_prep[0].unknown_datapoints = 66 rra[3].cf = "AVERAGE" rra[3].rows = 240 rra[3].pdp_per_row = 672 rra[3].xff = 5.0000000000e-01 rra[3].cdp_prep[0].value = 4.6278120560e+09 rra[3].cdp_prep[0].unknown_datapoints = 66 rra[4].cf = "AVERAGE" rra[4].rows = 370 rra[4].pdp_per_row = 5760 rra[4].xff = 5.0000000000e-01 rra[4].cdp_prep[0].value = 4.6278120560e+09 rra[4].cdp_prep[0].unknown_datapoints = 3810
#!/bin/sh
hosts=$1
metrics=$2
rate=$3
elapsed=0
sleepval=0
for ((;;)); do
epoch=`eval date +%s`
echo -n $epoch
for ((times=0;times<$hosts;times++)); do
# octet4=`expr 6 + $times`
# sourceip=`eval echo 192.168.40.$octet4`
for ((metric=0;metric<$metrics;metric++)); do
command=`eval echo gmetric --spoof=$times:host.$times
--name=metric.$metric --val=$epoch --units=time --type=uint16`
result=`eval $command`
sleep $sleepval
done
done
elapsed=`eval date +%s`
elapsed=`expr $elapsed - $epoch`
sleepval=`echo "scale=5; $sleepval +
((($hosts*$metrics)/$rate)-$elapsed)/($hosts*$metrics)" | bc`
echo -n " " $elapsed
echo " " $sleepval
done
gmetad.conf
Description: Binary data
gmond.conf
Description: Binary data
