Hi Henrik, On Tue, May 29, 2007 at 11:20:02PM +0200, Henrik Stoerner wrote: > On Thu, May 24, 2007 at 01:43:56PM -0500, Dave Plonka wrote: > > Note however, that such a configuration couldn't even do 100K > > RRD files without the patch to fadvise RANDOM to suppress readahead. > > I believe any post-2.6.5 Linux has the posix_fadvise behavior that > > the patch leverages. > > > > Also, Tobi has integrated the patch in the code he's testing. > > I "only" have about 40K RRD's with 5-minute update intervals, but the > machine had problems keeping up with this so I decided to try out your > patches. > > Since we're on RRDtool 1.0.49, though, I did a back-port of your > fadvise code to that version - very easy, but if anyone else > wants it drop me a note and I'll put the diff up for download. > > The result was that the memory utilisation for the buffer cache > dropped dramatically, as I'd expected.
Rather than trying to guess what's going on in the buffer cache, we could use fincore to determine how the RRD files are being cached: http://net.doit.wisc.edu/~plonka/fincore/ By running this periodically, one can see if things are going in the right direction or if, for instance, RRD files are competing with other things for buffer-cache. > However, the general I/O > activity did not go down - in fact, it increased by about 15-20%, > as measured by vmstat. With some more detail about what you saw, we could find what's really going on. reads or writes increased? If performance was so poor before such that it interfered with polling or graphing then it is possible for overall I/O to go up afterwards. Did the writes also increase or just the reads? By properly managing buffer-cache, it is possible to get RRD reads to near zero. Also, how long did it run before making this assessment (that I/O increased)? > This is a Linux 2.6.18 (Debian 4.0) system; the RRD files are on a > ReiserFS filesystem. I have no experience with reiser (just ext2 and ext3), but from what I've read about it, I wouldn't expect the file system to be a major performance concern here, since we really just want to use the buffer-cache effectively. > Re-coding the application that handles the RRD updates to cache them in > memory and bundle updates together turned out to be a lot more effective > way of tackling the problem. We're running Hobbit, not MRTG - I have no > idea if this would even be possible with MRTG. It is, and we've done that as well. From our research and experimentation the two methods are about equally as effective, and they are luckily complementary. That is, the application caching technique also benefits from the suppresses readahead that better matches RRD random I/O. My position on it is whether you have tens or low hundreds of thousands of files, one can assemble a Linux server (perhaps with multiple CPUs and Gigs of RAM) that can handle this level of performance. If that is not possible (for financial or other reasons) and you find that fadvise RANDOM alone results in insufficent performance on your system, then techniques which change the I/O pattern dramatically by application caching can be used in addition. Dave -- [EMAIL PROTECTED] http://net.doit.wisc.edu/~plonka/ Madison, WI _______________________________________________ rrd-developers mailing list [email protected] https://lists.oetiker.ch/cgi-bin/listinfo/rrd-developers
