> -----Original Message----- > From: [EMAIL PROTECTED] [mailto:rrd-users- > [EMAIL PROTECTED] On Behalf Of Ruttenberg, Tanya > Sent: Wednesday, January 25, 2006 11:20 AM > To: [email protected] > Subject: [rrd-users] Migrating RRD data to RDBMS > > We have a requirement to migrate data from RRD files to an RDBMS. > > My current idea is to do a daily export from the rrdfiles, parse the > export > file, and update the RDBMS with the resulting numbers. > > Is anyone out there doing this already? Suggestions?
I'm not going to say that what we're doing is optimal and I am very interested in other answers to this question but we're doing this for approximately 10,000 rrd files every 30 minutes, sticking the data into SQLServer. We've written a perl script that essentially does the following-- - Walk the file system (File::Find) looking for .rrd files, ignoring ones we don't care about - based on the directory and file name, query the SQLServer for the last timestamp for which we inserted non-NULL data - use RRDs::fetch to retrieve entries starting from the last timestamp in SQLServer until RRDs::last(rrdfile)-300 from the rrd file in question - insert the data into SQLServer - rinse, repeat We run multiple insert scripts in parallel, each looking at different parts of the file system. We keep data for ~600 days in the DB which now has about 100 billion rows. We run every 30 minutes because the RRD files are actually populated on other servers and are rsync'd over every 30 minutes. We could probably run on a much more frequent schedule if we didn't move the files around (NFS maybe?). In the event of some unforeseen failure of the insert process, we have plenty of other opportunities during the first 50 hours to obtain the 5 minute intervals we want from the first RRA. If we only ran once a day we'd have at most 1 other run to get that data without manual intervention. Hope that helps. -- Marc -- Unsubscribe mailto:[EMAIL PROTECTED] Help mailto:[EMAIL PROTECTED] Archive http://lists.ee.ethz.ch/rrd-users WebAdmin http://lists.ee.ethz.ch/lsg2.cgi
