Dear Jimmy and all I understand Jimmy ideas because my team is also developing similar features for ntop, namely - persistent data storage with mysql - decoupled web interface for plotting archived data (so that no need to run ntop while looking at history data)
Please correct me if I'm wrong. *What I think RRD and current NTOP can do for Jimmy's requirement.* - I think the old data will be stored in RRD even if NTOP crashes or restarts. You can still use RRD to plot history data before NTOP restarts. (Try clicking on magnifying glass icon next to each RRD graph. You will get a page that let you choose to view last 1 hr, last 2 hr, ... data. You can go back even to last year depending on how to set your RRD preferences.) - Since RRD normally plots data in rate. It doesn't matter if the NTOP crashes. However if you need raw data like in BYTE units, this variable will be reset when NTOP restarts. In this case you will need to monitor for restart. *What I think RRD and current NTOP cannot do and why persistent storage is needed.* - To obtain top X hosts during a specified period - To obtain top X applications during a specified period To do ranking we need to compare archived data of all hosts or all app. It's difficult to do with current RRD since each host and each app are stored separately. Anyway, my team has developed a plug-in for storing data into MySQL (currently it stores all data RRD stores + some more). *Can someone let me know how I can contribute plug-in to this project?* Panita On Fri, Jul 11, 2008 at 1:58 AM, Gary Gatten <[EMAIL PROTECTED]> wrote: > Do you have a router / routers that support netflow or sflow? nTop can > export those streams and they take up MUCH less storage than full packet > captures. If everyone comes back through a hub router you can easily > monitor usage from that point only. Many options here, but if > persistent storage is the key AND you need EVERYTHING the web interfaces > display (RRD doesn't store "everything") you may have to write the > scripts you mentioned. Have you looked at wget? The key thing is to > accurately identify the requirements and go from there - else you'll > waste a bunch of time gathering and massaging data that has little if > any value. > > What would be cool is if the web interface was decoupled from the data - > such that you could "easily" change between real time displays and those > of archived files. Of course you can look at "some" history easily and > some not so easy IF RRD is enabled, but if I want to view a dump file I > need to start an instance of ntop to view that dump? - I think? > > G > > PS: RRD does eat space if you have full detail logging in all > categories. > > > > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of > Jimmy > Sent: Thursday, July 10, 2008 11:52 AM > To: [email protected] > Subject: Re: [Ntop] nTop Multiple Instance Manager & Persistent > mySQLData Logger > > I really hope that nTop doesn't provide this functionality, and that I > haven't somehow missed it, or else I have wasted a considerable amount > of time! My main need is to have persistent storage. I can't lose data > if nTop crashes, or if the box goes down for some other reason. We also > need some of the data in SQL. The only way that I'm aware of to > accomplish this with nTop, as it is, would be to use the RRD plugin. As > I understand it - even if nTop is storing its data in RRD tables, it > still won't load that data back in if it is restarted. This means that > the graphs and other data found on the web front-end won't contain > anything logged before it was started. Then we'd still need to export > those tables to SQL. Also, in my brief experience with the RRD plugin, > the tables grew wildly out of control when logging hosts on a small > network. > > The other possibility would be to store the pcap traffic dumps, which > would waste a ton of space, BUT we'd be able to read them back in to > nTop to take a look at the different hosts, graphs, and other data - but > not across multiple runs of nTop. We'd have a set of dumps for each > time nTop was running, so if it was restarted three times, we'd have > three sets of traffic dumps (perhaps we could merge them?). > > The only solution that I can see to persistently store nTop's data, > without regard for nTop being restarted, is to write a collection of > scripts that monitor the nTop service(s), regularly (currently, hourly) > dump its counters, and generate statistics and graphs similar to the > embedded front-end, but using our dumped data. This sacrifices the > possibility of having live data, BUT in a situation where you're looking > for network trends, and per-host totals, that's really not very > important anyway. > > If there is a simpler way to do this, please let me know ASAP! > > When I have some more free time, I'd be happy to write some > documentation. I've become very well acquainted with the software over > the past few weeks, and could write a fairly in-depth paper on the > different command-line switches, what nTop logs, and what nTop does and > does not do. > > -Jimmy > > > On Tue, 2008-07-08 at 17:30 -0500, Gary Gatten wrote: > > Sounds interesting, but I think what your client wants to accomplish > can > > be done without writing anything. I'll read your requirements again > > more carefully and see what's up. nTop is pretty flexible, but would > > benefit from some written case studies, how to's, etc. I've been > > meaning to write some doc myself - especially pertaining to netflow - > > but haven't :( > > > > G > > > > > > -----Original Message----- > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf > Of > > Jimmy > > Sent: Thursday, July 03, 2008 1:33 PM > > To: [email protected] > > Subject: [Ntop] nTop Multiple Instance Manager & Persistent mySQL Data > > Logger > > > > (I accidentally sent this from an address that's not subscribed to > this > > list, so I assume it was discarded - I apologize if it wasn't, and > this > > is the second time you're receiving this.) > > > > Hello everyone - I'm new to this list, so I'm not sure if I'm posting > > this to the correct one. I wanted this information to reach > > non-developers, so I figured it would be appropriate to post it here. > > > > I'm in the process of writing a collection of PHP scripts designed to > > monitor various instances of nTop, perform hourly dumps of ntop's > > counters, and tie in seamlessly with a web front-end that uses > Google's > > charts API to display some pretty charts and graphs of the data it has > > collected. All of the data is stored in mySQL tables. > > > > Here is a short list of features: > > *Monitor different instances of nTop (is the daemon running?) > > *Automatically restart, and/or e-mail someone about, nTop instances > that > > go down unexpectedly. > > *Persistently store data about each interface and each host, on an > > hourly basis, and account for the counters being reset due to ntop > being > > restarted. > > *The front-end displays most of the same data that the ntop embedded > > front-end displays, except for data that ntop doesn't dump (such as > port > > distribution, per-host port usage, recent/active per-host sessions, > > etc.). (In the future, I will write some scripts that extract certain > > data from nTop's front-end, but that's a whole project in itself, so > > it's pretty much dead last on my list.) > > > > Those are the main features - but the project is not complete, and I > am > > sure I will add more functionality before I begin using it regularly, > > and before it is released publicly. > > > > I decided to write these scripts because I was asked to perform some > > traffic and bandwidth utilization analysis for a mid-size company. > nTop > > is great software, but I needed to be able to permanently store its > data > > in a commonly used format, and most importantly, manage multiple > > instances. My client has multiple sites spread across NYC, all of the > > traffic goes to the central office, and and to their T3. They want > > statistics for each site, individually. This script suite is designed > > for this exact purpose - to manage multiple instances of nTop, > > permanently store the data that it accumulates, and do this for a week > > or two week long study. There will be such a large amount of data > > accumulated throughout this study, that it was simply not reasonable > to > > have nTop dump its data to RRD tables, then manually graph different > > data. This means I'd have to write a front-end that pulled data from > > the RRD tables and displayed them, meaning it'd be easier, quicker, > and > > more space-friendly for me to have nTop dump the data, and have my > > scripts pick and choose what to store permanently. > > > > I just wanted to give everyone a heads up, in case you might find this > > at all useful. When this project is complete, and I start using these > > scripts for the actual study, I will release what I have, hopefully > > along with some decent documentation. > > > > Feel free to e-mail me with any questions. If anyone knows of an > > appropriate location, associated with nTop, for me to post these > > scripts, I'd be happy to do so. > > > > Thanks, and have a great day! > > > > -Jimmy Ryan > > > > > > _______________________________________________ > > Ntop mailing list > > [email protected] > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > > > > > > > <font size="1"> > > <div style='border:none;border-bottom:double windowtext > 2.25pt;padding:0in 0in 1.0pt 0in'> > > </div> > > "This email is intended to be reviewed by only the intended recipient > > and may contain information that is privileged and/or confidential. > > If you are not the intended recipient, you are hereby notified that > > any review, use, dissemination, disclosure or copying of this email > > and its attachments, if any, is strictly prohibited. If you have > > received this email in error, please immediately notify the sender by > > return email and delete this email from your system." > > </font> > > > > _______________________________________________ > > Ntop mailing list > > [email protected] > > http://listgateway.unipi.it/mailman/listinfo/ntop > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > <font size="1"> > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in > 0in 1.0pt 0in'> > </div> > "This email is intended to be reviewed by only the intended recipient > and may contain information that is privileged and/or confidential. > If you are not the intended recipient, you are hereby notified that > any review, use, dissemination, disclosure or copying of this email > and its attachments, if any, is strictly prohibited. If you have > received this email in error, please immediately notify the sender by > return email and delete this email from your system." > </font> > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop >
_______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
