Hi Jesse, On Fri, Aug 16, 2013 at 06:49:25PM +0930, Jesse Reynolds wrote: > Sounds too good to be true :-)
collectd's "rrdtool" plugin and RRDCacheD use the same basic principle: Cache updates to RRD files in memory and write multiple updates at once. Due to the way RRD files are laid out, this will result in a sequential write. On spinning disks, updated to RRD files are dominated by disk seeks. By batching updates, you get _way_ better performance. As in, updating one metric vs. 32 metrics boils down to the same number of disk seeks. In contrast to this, the amount of data being written, at the scale of metrics written to RRD files, does not matter. Whether you write one metric (8 bytes) or 32 metrics (256 bytes, 1-2 blocks) or 1024 metrics (8 kByte, 32-33 blocks) doesn't make a difference in disk latency. On SSDs, "seeks" don't exist in the same way they do on spinning media, but writing sequentially still has benefits. My understanding of SSDs is based on hear-say, so please take this with a grain of salt. As far as I know, one bit transition, e.g. from 1 to 0 is easy for Flash, the other way around, i.e. from 0 to 1, is _not_. As in, you can only reset an entire block to all 1's and then flip individual bits to 0. These blocks are quite large, in the 4 kByte ballpark. Since this is slow, the entire block is usually written to somewhere else and the previous copy blanked in the background. Again, sequential writes are a big advantage, but for entirely different reasons. Although the principle is the same, there are differences between the "rrdtool" plugin and RRDCacheD: The plugin can write at a configured rate (e.g. 50 writes per second), the daemon can write a journal and does not need to write all state to disk when shutting down. The biggest advantage for RRDCacheD, though, is that it allows you to restart collectd without flushing everything to disk. > Is the scheduling improved somehow? Is it more efficient with writing > the RRD files? It seems that it will still need to update the RRD > files as the rrd plugin does now, so not sure how this means there'll > be less demands placed on IO. I/O wise, the two should be the same. If not, I'd love to know why and back-port any performance improvements to the "rrdtool" plugin. Back to the problem at hand, I have a theory: I noticed that there is an increase in used disk space whenever the performance degrades. That is usually caused by new RRD files being created. This used to block the "rrdtool" plugin, leading to less normal updates per second being performed. Add the long wait for file creation and you have an explanation for the average wait time increasing. You can try the "CreateFilesAsync" option which was added to the RRDtool plugin in version 5.3.0 to fix this issue: <http://collectd.org/documentation/manpages/collectd.conf.5.shtml#createfilesasync_false_true2> See also: <https://github.com/collectd/collectd/pull/262>. Hope this helps :) Best regards, —octo -- collectd – The system statistics collection daemon Website: http://collectd.org Google+: http://collectd.org/+ GitHub: https://github.com/collectd Twitter: http://twitter.com/collectd
signature.asc
Description: Digital signature
_______________________________________________ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd