Re: [collectd] RRDs on SSD - performance degredation [resend without images]
Hello, On Tue, Sep 24, 2013 at 05:51:57PM +0200, Mariusz Gronczewski wrote: You *could* just bump up flush timers in rrdtool plugin like that: As an alternative, the WritesPerSecond might be interesting, because the number of writes performed no longer depends on the number of metrics on your system. Best regards, —octo -- collectd – The system statistics collection daemon Website: http://collectd.org Google+: http://collectd.org/+ GitHub: https://github.com/collectd Twitter: http://twitter.com/collectd signature.asc Description: Digital signature ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd
Re: [collectd] RRDs on SSD - performance degredation [resend without images]
On 8/16/2013 4:14 AM, Cyril Feraudet wrote: rrdcached is a daemon and a Collectd plugin who replace rrdtool plugin. There is no need to modify your architecture. Just install rrdcached and configure rrdcached Collectd plugin to work with. Before : LoadPlugin rrdtool Plugin rrdtool DataDir /var/lib/collectd/rrd /Plugin After : LoadPlugin rrdcached Plugin rrdcached DaemonAddress unix:/var/run/rrdcached/rrdcached.sock DataDir /var/lib/collectd/rrd CreateFiles true CollectStatistics true /Plugin Cyril How do you deal with the options: RRARows 2 # PDPs = timespan / (stepsize * rrarows) RRATimespan 32 # 3.7 days (16s) RRATimespan 160 # 18.5 days (80s) RRATimespan 800 # 92.6 days (400s) RRATimespan 8000 # 926 days / 30.5 months / 2.54 years (4000s) Which are normally placed inside of Plugin rrdtool/Plugin? ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd
Re: [collectd] RRDs on SSD - performance degredation [resend without images]
Hi Jesse, On Fri, Aug 16, 2013 at 06:49:25PM +0930, Jesse Reynolds wrote: Sounds too good to be true :-) collectd's rrdtool plugin and RRDCacheD use the same basic principle: Cache updates to RRD files in memory and write multiple updates at once. Due to the way RRD files are laid out, this will result in a sequential write. On spinning disks, updated to RRD files are dominated by disk seeks. By batching updates, you get _way_ better performance. As in, updating one metric vs. 32 metrics boils down to the same number of disk seeks. In contrast to this, the amount of data being written, at the scale of metrics written to RRD files, does not matter. Whether you write one metric (8 bytes) or 32 metrics (256 bytes, 1-2 blocks) or 1024 metrics (8 kByte, 32-33 blocks) doesn't make a difference in disk latency. On SSDs, seeks don't exist in the same way they do on spinning media, but writing sequentially still has benefits. My understanding of SSDs is based on hear-say, so please take this with a grain of salt. As far as I know, one bit transition, e.g. from 1 to 0 is easy for Flash, the other way around, i.e. from 0 to 1, is _not_. As in, you can only reset an entire block to all 1's and then flip individual bits to 0. These blocks are quite large, in the 4 kByte ballpark. Since this is slow, the entire block is usually written to somewhere else and the previous copy blanked in the background. Again, sequential writes are a big advantage, but for entirely different reasons. Although the principle is the same, there are differences between the rrdtool plugin and RRDCacheD: The plugin can write at a configured rate (e.g. 50 writes per second), the daemon can write a journal and does not need to write all state to disk when shutting down. The biggest advantage for RRDCacheD, though, is that it allows you to restart collectd without flushing everything to disk. Is the scheduling improved somehow? Is it more efficient with writing the RRD files? It seems that it will still need to update the RRD files as the rrd plugin does now, so not sure how this means there'll be less demands placed on IO. I/O wise, the two should be the same. If not, I'd love to know why and back-port any performance improvements to the rrdtool plugin. Back to the problem at hand, I have a theory: I noticed that there is an increase in used disk space whenever the performance degrades. That is usually caused by new RRD files being created. This used to block the rrdtool plugin, leading to less normal updates per second being performed. Add the long wait for file creation and you have an explanation for the average wait time increasing. You can try the CreateFilesAsync option which was added to the RRDtool plugin in version 5.3.0 to fix this issue: http://collectd.org/documentation/manpages/collectd.conf.5.shtml#createfilesasync_false_true2 See also: https://github.com/collectd/collectd/pull/262. Hope this helps :) Best regards, —octo -- collectd – The system statistics collection daemon Website: http://collectd.org Google+: http://collectd.org/+ GitHub: https://github.com/collectd Twitter: http://twitter.com/collectd signature.asc Description: Digital signature ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd
Re: [collectd] RRDs on SSD - performance degredation [resend without images]
On 16/08/2013, at 5:16 PM, Cyril Feraudet colle...@feraudet.com wrote: Do you use rrdcached ? If not, use it and all you i/o problem will disappear. Sounds too good to be true :-) Is the scheduling improved somehow? Is it more efficient with writing the RRD files? It seems that it will still need to update the RRD files as the rrd plugin does now, so not sure how this means there'll be less demands placed on IO. We're planning on switching to opentsdb but that may be a little way off. I have looked at rrdcached. With switching to rrdcached, can you do this while keeping the RRD files in place? Or is there a migration process? Cheers Jesse ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd
Re: [collectd] RRDs on SSD - performance degredation [resend without images]
Try it and you'll see. rrdcached write to disk on demand and at the latest when the time specified is out. I/O may be decreased by 10 or more depending on the interval on collectd side and the '-w' option on rrdcached side. There is no migration to plan. Just start properly rrdcached and configure collectd, ex : /usr/bin/rrdcached -w 1800 -z 1800 -t 8 -p /var/run/rrdcached.pid -l unix:/var/run/rrdcached/rrdcached.sock -j /var/lib/rrdcached/journal And make sure the user used to run rrdcached is able to write rrd. One more thing : rrdtool graph command need to know the rrdcached socket to flush data before graph them, ex: rrdtool graph --daemon=/var/run/rrdcached/rrdcached.sock -t Cyril Le 2013-08-16 11:19, Jesse Reynolds a écrit : On 16/08/2013, at 5:16 PM, Cyril Feraudet colle...@feraudet.com wrote: Do you use rrdcached ? If not, use it and all you i/o problem will disappear. Sounds too good to be true :-) Is the scheduling improved somehow? Is it more efficient with writing the RRD files? It seems that it will still need to update the RRD files as the rrd plugin does now, so not sure how this means there'll be less demands placed on IO. We're planning on switching to opentsdb but that may be a little way off. I have looked at rrdcached. With switching to rrdcached, can you do this while keeping the RRD files in place? Or is there a migration process? Cheers Jesse ___ collectd mailing list collectd@verplant.org http://mailman.verplant.org/listinfo/collectd
