On 1/10/06, Ruttenberg, Tanya <[EMAIL PROTECTED]> wrote: > This is an interesting dilemma.
Have to agree with you here... > I've had a lot of thoughts about this since I started replying to your mail. > > One of the main features of rrdtool is its ability to consolidate the data > over a period of time. Or age out the data if you decide not to use RRA's. > You have to decide ahead of time the maximum amount of data you are ever > going to want to store there. This is where option #1 comes in handy, you can generate an rrd of the required size on the fly. > It seems from your email that you are planning to WORK AROUND that feature > and use rrd files as an alternative to your mysql database. For instance, > you want to be able to see a graph of 90 days of data that reflects the data > in your mysql database. Is this correct? Never thought about it being a work around, but it appears so. Yip, all the data resides in a MySQL database. > If this is true, and you decide to use pre-created rrd files, it seems to me > you will have to create rrd files large enough to store the maximum amount > of data anyone could ever want to see a graph of. If this is a possibility > in your environment, then I think suggestion 2 is the way to go. Or even > better, if you can save the pmacct data in the rrd files as you collect > it--at the same time you store it in your rdbms. I don't know how you get > the data into mysql (not familiar with pmacct), but if it is put there via a > script you wrote, you can also write into it an rrdtool "adapter" that puts > the same data into rrd files at the time it is collected. pmacct does the dirty work for me here, it saves the accounting directly to MySQL. I can however configure pmacct to execute a script upon each write to the database, so I can update the rrd's like that. pmacct is a great tool for collecting data for network accounting purposes, check out http://www.ba.cnr.it/~paolo/pmacct/. Using some smart tricks you can populate rrd's directly using it's memory plugin (instead of MySQL), but I require the MySQL route here... > If you cannot predict how much data someone is going to need (for instance > you never know if someone is going to want to view a graph with 2 years' > worth of data) then you have no idea how large to create your rrdfiles and > you have to somehow go with #1. Though I don't see how to get around the > performance issue there. Agreed... > There is not some other tool that extracts data from a database and > generates a graph directly from this data? Though I agree rrdtool has > excellent graphing ability so I can see why you would want to leverage that. The reporting system is coded in PHP, and there are a good couple of PHP classes that generate all kinds of statistical graphs, charts, etc.. The one I've found the best so far is a library called jpGraph. But it chokes on creating a graph with so much data and the memory consumption in PHP is tremendous when doing this. This triggered my investigations into an alternative, hence rrd. > Anyway, I'd be interested to hear how you resolve this. I'm digging around the cacti code to see how they handle it, but it seems that option #2 is definitely the way to go here... > Tanya Ruttenberg - RSIS Contractor > OTSO/DNE/NMPEB > [EMAIL PROTECTED] > 410-965-9605 Kenneth > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Kenneth Kalmer > Sent: Tuesday, January 10, 2006 6:15 AM > To: [email protected] > Subject: [rrd-users] Strategies for data storage and graphing > > Guys > > This is my first post to the list, please excuse if me if it's not up to > scratch... > > I currently collect network accounting data using pmacct and save it to > MySQL. Due to the nature of the network we need to keep a massive amount of > data for various reporting. I've been playing with rrdtool and a sample of > the data with great results. > > I'd like to know what would be the best way to populate the rrd's from the > data in the database to create graphs. I know there are plenty of ways, but > speed is number one here. I'm also positive that I'm not first one facing > this problem... > > Currently I'm faced with two possibilities, and would like any suggestions > on better ways of doing this. > > 1) Create the rrd, populate and graph in one smooth step. Works fine, but to > populate 90 days worth of 300 second step data can take up to 5 minutes... > Not desirable... > > 2) At set intervals, simply populate pre-created rrd's with the new data > since the last update, and when needed to graph add the remaining data > on-demand... Keeping track of the updates won't be difficult... > Haven't built this yet, but in theory it should yield much better > performance. > > It boils down to needing rrd's excellent graphing capabilities, as well as > blistering speed when generating reports. This must be the same problem the > developers of cacti faced with all the data in MySQL, and rrdtools > graphing... Speaking of which, I'm about to tear up the cacti code looking > for insight, but any advise would be appreciated... > > Best > > -- > > Kenneth Kalmer > [EMAIL PROTECTED] > > [EMAIL PROTECTED] stats > http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpage&username=kenneth% > 2Ekalmer > > -- > Unsubscribe mailto:[EMAIL PROTECTED] > Help mailto:[EMAIL PROTECTED] > Archive http://lists.ee.ethz.ch/rrd-users > WebAdmin http://lists.ee.ethz.ch/lsg2.cgi > -- Kenneth Kalmer [EMAIL PROTECTED] [EMAIL PROTECTED] stats http://fah-web.stanford.edu/cgi-bin/main.py?qtype=userpage&username=kenneth%2Ekalmer -- Unsubscribe mailto:[EMAIL PROTECTED] Help mailto:[EMAIL PROTECTED] Archive http://lists.ee.ethz.ch/rrd-users WebAdmin http://lists.ee.ethz.ch/lsg2.cgi
