Hey All - I've been messing around with getting data from collectd into something other than RRD (in my case it is carbon/graphite) and one of the easiest ways I could see to do this was to have the write_http plugin PUT the data in JSON format to a central location, have the listening server parse the JSON and then send it along to carbon via an AMQP cluster. Immediately I noticed that there were gaps in the data that was being graphed (anywhere from 5 to 20 seconds or so) which struck me as fairly odd, especially considering that all metrics from the host for that time period was not showing up (see attached screenshot).
Initially I thought that this was caused somewhere along the line by the pieces I had written, so I set about optimizing the code I had written (which really isn't a bad thing) and checking into what carbon and the AMQP were doing. After a while of chasing down dead end leads I took a look at exactly what collectd was doing. I've taken a look at collectd 4.10.3 and 5.0.0 and all of them appear to exhibit the same behavior: unless the logs are lying, once the write_http buffer fills up and sends the data off, no information is gathered by collectd. I've attached the debug logs from both host1 (collectd 4.10.3) and host2 (collectd 5.0.0). If the log file is accurate, is this the expected behavior of the write_http plugin? I've also attached our minimal collectd.conf, which gathers load data once per second, and just for testing is only gathering load info. Thank you! -pat
<<attachment: Screen shot 2011-09-21 at 11.07.02 AM.png>>
[2011-09-21 10:58:52] plugin_read_thread: Handling `load'.
[2011-09-21 10:58:52] plugin_dispatch_values: time = 1316627932; interval = 1;
host = host1.colo.eharmony.com; plugin = load; plugin_instance = ; type = load;
type_instance = ;
[2011-09-21 10:58:52] uc_update: host1.colo.eharmony.com/load/load: ds[0] =
0.070000
[2011-09-21 10:58:52] uc_update: host1.colo.eharmony.com/load/load: ds[1] =
0.090000
[2011-09-21 10:58:52] uc_update: host1.colo.eharmony.com/load/load: ds[2] =
0.020000
[2011-09-21 10:58:52] plugin: plugin_write: Writing values via write_http.
[2011-09-21 10:58:52] format_json: values_to_json: buffer = [0.07,0.09,0.02];
[2011-09-21 10:58:52] format_json: dstypes_to_json: buffer =
["gauge","gauge","gauge"];
[2011-09-21 10:58:52] format_json: dsnames_to_json: buffer =
["shortterm","midterm","longterm"];
[2011-09-21 10:58:52] format_json: value_list_to_json: buffer =
,{"values":[0.07,0.09,0.02],"dstypes":["gauge","gauge","gauge"],"dsnames":["shortterm","midterm","longterm"],"time":1316627932,"interval":1,"host":"host1.colo.eharmony.com","plugin":"load","plugin_instance":"","type":"load","type_instance":""};
[2011-09-21 10:58:52] write_http plugin:
<http://metrics.colo.eharmony.com:8080/post-collectd> buffer 3912/4096
(95.5078%)
[2011-09-21 10:58:52] plugin_read_thread: Effective interval of the load plugin
is 1.000000000.
[2011-09-21 10:58:52] plugin_read_thread: Next read of the load plugin at
1316627933.311803000.
[2011-09-21 10:58:53] plugin_read_thread: Handling `load'.
[2011-09-21 10:58:53] plugin_dispatch_values: time = 1316627933; interval = 1;
host = host1.colo.eharmony.com; plugin = load; plugin_instance = ; type = load;
type_instance = ;
[2011-09-21 10:58:53] uc_update: host1.colo.eharmony.com/load/load: ds[0] =
0.070000
[2011-09-21 10:58:53] uc_update: host1.colo.eharmony.com/load/load: ds[1] =
0.090000
[2011-09-21 10:58:53] uc_update: host1.colo.eharmony.com/load/load: ds[2] =
0.020000
[2011-09-21 10:58:53] plugin: plugin_write: Writing values via write_http.
[2011-09-21 10:58:53] format_json: values_to_json: buffer = [0.07,0.09,0.02];
[2011-09-21 10:58:53] format_json: dstypes_to_json: buffer =
["gauge","gauge","gauge"];
[2011-09-21 10:58:53] format_json: dsnames_to_json: buffer =
["shortterm","midterm","longterm"];
[2011-09-21 10:58:53] write_http plugin: wh_flush_nolock: timeout = -1;
send_buffer_fill = 3912;
[2011-09-21 10:58:55] uc_check_timeout: host1.colo.eharmony.com/load/load is
missing but ``uninteresting''
[2011-09-21 10:59:23] format_json: values_to_json: buffer = [0.07,0.09,0.02];
[2011-09-21 10:59:23] format_json: dstypes_to_json: buffer =
["gauge","gauge","gauge"];
[2011-09-21 10:59:23] format_json: dsnames_to_json: buffer =
["shortterm","midterm","longterm"];[2011-09-20 22:09:17] plugin_read_thread: Handling `load'.
[2011-09-20 22:09:17] plugin_dispatch_values: time = 1316581757.237; interval =
1.000; host = host2.colo.eharmony.com; plugin = load; plugin_instance = ; type
= load; type_instance = ;
[2011-09-20 22:09:17] uc_update: host2.colo.eharmony.com/load/load: ds[0] =
0.000000
[2011-09-20 22:09:17] uc_update: host2.colo.eharmony.com/load/load: ds[1] =
0.000000
[2011-09-20 22:09:17] uc_update: host2.colo.eharmony.com/load/load: ds[2] =
0.000000
[2011-09-20 22:09:17] plugin: plugin_write: Writing values via write_http.
[2011-09-20 22:09:17] format_json: values_to_json: buffer = [0,0,0];
[2011-09-20 22:09:17] format_json: dstypes_to_json: buffer =
["gauge","gauge","gauge"];
[2011-09-20 22:09:17] format_json: dsnames_to_json: buffer =
["shortterm","midterm","longterm"];
[2011-09-20 22:09:17] format_json: value_list_to_json: buffer =
,{"values":[0,0,0],"dstypes":["gauge","gauge","gauge"],"dsnames":["shortterm","midterm","longterm"],"time":1413668897460837414,"interval":1.000,"host":"host2.colo.eharmony.com","plugin":"load","plugin_instance":"","type":"load","type_instance":""};
[2011-09-20 22:09:17] write_http plugin:
<http://metrics.colo.eharmony.com:8080/post-collectd> buffer 4016/4096
(98.0469%)
[2011-09-20 22:09:17] plugin_read_thread: Effective interval of the load plugin
is 1.000000000.
[2011-09-20 22:09:17] plugin_read_thread: Next read of the load plugin at
1316581758.234909999.
[2011-09-20 22:09:18] plugin_read_thread: Handling `load'.
[2011-09-20 22:09:18] plugin_dispatch_values: time = 1316581758.236; interval =
1.000; host = host2.colo.eharmony.com; plugin = load; plugin_instance = ; type
= load; type_instance = ;
[2011-09-20 22:09:18] uc_update: host2.colo.eharmony.com/load/load: ds[0] =
0.000000
[2011-09-20 22:09:18] uc_update: host2.colo.eharmony.com/load/load: ds[1] =
0.000000
[2011-09-20 22:09:18] uc_update: host2.colo.eharmony.com/load/load: ds[2] =
0.000000
[2011-09-20 22:09:18] plugin: plugin_write: Writing values via write_http.
[2011-09-20 22:09:18] format_json: values_to_json: buffer = [0,0,0];
[2011-09-20 22:09:18] format_json: dstypes_to_json: buffer =
["gauge","gauge","gauge"];
[2011-09-20 22:09:18] format_json: dsnames_to_json: buffer =
["shortterm","midterm","longterm"];
[2011-09-20 22:09:18] write_http plugin: wh_flush_nolock: timeout = 0.000;
send_buffer_fill = 4016;
[2011-09-20 22:09:28] format_json: values_to_json: buffer = [0,0,0];
[2011-09-20 22:09:28] format_json: dstypes_to_json: buffer =
["gauge","gauge","gauge"];
[2011-09-20 22:09:28] format_json: dsnames_to_json: buffer =
["shortterm","midterm","longterm"];
collectd.conf
Description: collectd.conf
_______________________________________________ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd
