On Fri, 2005-03-11 at 05:18, Ramon Bastiaans wrote: > >To save the data in case of a system crash, I just patched the gmetad > >init script to backup and restore the rrds with tar when it is stopped > >and started, then use a daily cronjob to restart gmetad every night. I > >stop gmetad before backing up otherwise tar complains that one or more > >files has changed while it was being read. I have attached the init > >patch that I use in case you are interested. > > > >~Jason > > > > > So if there is a kernel panic or similar (lets hope not) you only have > the data from your last backup, which was on startup or at midnight, > right? Aren't you loosing big timeframes in case a crash occures? I have > been thinking about a ramdisk too, but are somewhat reluctant on the > data loss in case of a crash. How to maintain a recent (backup) copy to > disk, so that as little data as possible is lost on a crash?
Basically I don't worry about the possibility of losing a small amount of data because of the way rrdtool archives it. With the default update interval of 15 seconds, ganglia creates rrds to save one hour worth of raw data, one day worth of 6-minute averaged data, one week worth of 42-minute averaged data, one month worth of 168-minute averaged data and 1 year worth of 24-hour averaged data. See ganglia-3.0.0/gmetad/rrd_helpers.c for the rrd archive definitions: argv[argc++] = "RRA:AVERAGE:0.5:1:240"; argv[argc++] = "RRA:AVERAGE:0.5:24:240"; argv[argc++] = "RRA:AVERAGE:0.5:168:240"; argv[argc++] = "RRA:AVERAGE:0.5:672:240"; argv[argc++] = "RRA:AVERAGE:0.5:5760:370"; With a fatal crash (which is rare to begin with) right before the nightly backup is made, you would lose at most 24 hours worth of data. This would be negligible in the yearly and monthly archives with only 1 and ~8 unknown values respectively. It would obviously be more noticeable in the weekly, monthly and hourly archives, but they don't exist for very long anyway so why worry about them when it comes to long term storage and recovery backup? > I am also interested because for a project of mine I have to archive the > rrd data for the historical tracking of compute jobs and their > performance. However, copy'ing 135 Mb worth of rrd files every hour - in > less than 15 seconds (metric interval) - is hardly doable. Using a > ramdisk would certainly speed up things. I am tempted though to write an > other tool who archives the data directly from the multicast channel, > but this would mean even more disk access (and load). > > - Ramon. If you really want long term storage of the raw or nearly raw data then rrdtool is probably not the right tool to use. You would be better off writing your own ganglia frontend client that would collect the xml data from gmetad at the interval you need, parse it and store it into some other database or archive. This could also be done from another computer so it would have a negligible impact on the gmetad host. ~Jason -- /------------------------------------------------------------------\ | Jason A. Smith Email: [EMAIL PROTECTED] | | Atlas Computing Facility, Bldg. 510M Phone: (631)344-4226 | | Brookhaven National Lab, P.O. Box 5000 Fax: (631)344-7616 | | Upton, NY 11973-5000 | \------------------------------------------------------------------/
