Arjun, Ramon, my numbers look a bit different, but equally disturbing:
lets assume 300 hosts with 36 metrics. I would not look at the RRD format, but just store samples as they come from gmond. That means we have 300x36 values per sample. About 12000. Now lets assume the same sample rate as the 1hour resolution in RRD. That gives 300x36x240. about 2.6 mio values per hour. About 62.3 mio values per day About 22.8e9 values per year That is a lot of capacity and a lot of needed performance. Of course, a lot of the metrics in Ganglia are not that interesting to most people, or do not need the 15sec resolution. Cheers Martin --- Ramon Bastiaans <[EMAIL PROTECTED]> wrote: > Arjun, > > I think performance of this database will be a HUGE issue, depending > on > how many metrics/hosts/clusters and timespan that you wish to store. > > Now please correct me if I'm wrong, but lets make a estimated > calcution > on what is going to be stored in the database. > Let's make a few assumptions; you are only storing the default > Ganglia > metrics and no custom/extra metrics and you have a cluster of 300 > hosts. > Also, you don't archive any values and you use the same graph > resolution/value scheme as used by the RRD's from gmetad (the same > static amount of rotating values per resolution: hours, days, weeks, > months, years). > > Metrics per host: 36 > Hosts per cluster: 300 > > Now let's make a quick estimate on how many values you are going to > store in mySql. > Ganglia uses about 240 rows per resolution and 370 rows for the year > summary, this is 1330 rows per metric, per host. > > 1330 * 36 * 300 = 14364000 values. > > This comes down to allmost 15 million values in your database, when > using the same style of value storing as currently done by Ganglia in > RRD's. > > Now if you add: > - extra hosts: 1330 * 36 = 47880 values/host > - extra metrics: 1330 * 300 = 399000 values/metric > > I don't know your particular setup, but here at SARA we monitor about > > 1800 machines in total with more than 50 metrics per host. > A quick estimate would come to 120 million values in the database. > > Now imagine quering/selecting from such a database.... > The performance would seem hell to me, making it totally unusable > from > the web environment where you want the values. > > Also take into account that normally on the web frontend, graphs are > generated by RRD itself. But now you are using a database, so if you > want RRDTool to draw the graphs for you, you need to convert your > database values back to some sort of RRD format. This means a lot of > query'ing and converting of those values, each time a > (host/metric/whatever) graph is requested. This would also require > additional hacks (or changes) to the web frontend code as well. > > I think you might need insane if not impossible hardware to support > such > a (database) setup, but anyone correct me if I'm wrong. > > Kind regards, > - Ramon. > > Arjun wrote: > > In my case the monitoring db will be on a separate machine along > with > > gmetad. I'm monitoring a cluster so can have a > > separate (external) machine to store data on so I guess this will > not > > be a performace bottleneck if I have a DB like > > MySql to store and retreive data. > > > > thanks > > Arjun > > > -- > ing. R. Bastiaans HPC - Systems Programmer > > SARA - Computing and Networking Services > Kruislaan 415 PO Box 194613 > 1098 SJ Amsterdam 1090 GP Amsterdam > Tel. +31 (0) 20 592 3000 Fax. +31 (0) 20 668 3167 > --- > There are really only three types of people: > > Those who make things happen, those who watch things happen > and those who say, "What happened?" > > > > ------------------------------------------------------- > This SF.Net email is sponsored by xPML, a groundbreaking scripting > language > that extends applications into web and mobile media. Attend the live > webcast > and join the prime developer group breaking into this new coding > territory! > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642 > _______________________________________________ > Ganglia-developers mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/ganglia-developers > > ------------------------------------------------------ Martin Knoblauch email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de
