Do you have a router / routers that support netflow or sflow? nTop can export those streams and they take up MUCH less storage than full packet captures. If everyone comes back through a hub router you can easily monitor usage from that point only. Many options here, but if persistent storage is the key AND you need EVERYTHING the web interfaces display (RRD doesn't store "everything") you may have to write the scripts you mentioned. Have you looked at wget? The key thing is to accurately identify the requirements and go from there - else you'll waste a bunch of time gathering and massaging data that has little if any value.
What would be cool is if the web interface was decoupled from the data - such that you could "easily" change between real time displays and those of archived files. Of course you can look at "some" history easily and some not so easy IF RRD is enabled, but if I want to view a dump file I need to start an instance of ntop to view that dump? - I think? G PS: RRD does eat space if you have full detail logging in all categories. -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jimmy Sent: Thursday, July 10, 2008 11:52 AM To: [email protected] Subject: Re: [Ntop] nTop Multiple Instance Manager & Persistent mySQLData Logger I really hope that nTop doesn't provide this functionality, and that I haven't somehow missed it, or else I have wasted a considerable amount of time! My main need is to have persistent storage. I can't lose data if nTop crashes, or if the box goes down for some other reason. We also need some of the data in SQL. The only way that I'm aware of to accomplish this with nTop, as it is, would be to use the RRD plugin. As I understand it - even if nTop is storing its data in RRD tables, it still won't load that data back in if it is restarted. This means that the graphs and other data found on the web front-end won't contain anything logged before it was started. Then we'd still need to export those tables to SQL. Also, in my brief experience with the RRD plugin, the tables grew wildly out of control when logging hosts on a small network. The other possibility would be to store the pcap traffic dumps, which would waste a ton of space, BUT we'd be able to read them back in to nTop to take a look at the different hosts, graphs, and other data - but not across multiple runs of nTop. We'd have a set of dumps for each time nTop was running, so if it was restarted three times, we'd have three sets of traffic dumps (perhaps we could merge them?). The only solution that I can see to persistently store nTop's data, without regard for nTop being restarted, is to write a collection of scripts that monitor the nTop service(s), regularly (currently, hourly) dump its counters, and generate statistics and graphs similar to the embedded front-end, but using our dumped data. This sacrifices the possibility of having live data, BUT in a situation where you're looking for network trends, and per-host totals, that's really not very important anyway. If there is a simpler way to do this, please let me know ASAP! When I have some more free time, I'd be happy to write some documentation. I've become very well acquainted with the software over the past few weeks, and could write a fairly in-depth paper on the different command-line switches, what nTop logs, and what nTop does and does not do. -Jimmy On Tue, 2008-07-08 at 17:30 -0500, Gary Gatten wrote: > Sounds interesting, but I think what your client wants to accomplish can > be done without writing anything. I'll read your requirements again > more carefully and see what's up. nTop is pretty flexible, but would > benefit from some written case studies, how to's, etc. I've been > meaning to write some doc myself - especially pertaining to netflow - > but haven't :( > > G > > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of > Jimmy > Sent: Thursday, July 03, 2008 1:33 PM > To: [email protected] > Subject: [Ntop] nTop Multiple Instance Manager & Persistent mySQL Data > Logger > > (I accidentally sent this from an address that's not subscribed to this > list, so I assume it was discarded - I apologize if it wasn't, and this > is the second time you're receiving this.) > > Hello everyone - I'm new to this list, so I'm not sure if I'm posting > this to the correct one. I wanted this information to reach > non-developers, so I figured it would be appropriate to post it here. > > I'm in the process of writing a collection of PHP scripts designed to > monitor various instances of nTop, perform hourly dumps of ntop's > counters, and tie in seamlessly with a web front-end that uses Google's > charts API to display some pretty charts and graphs of the data it has > collected. All of the data is stored in mySQL tables. > > Here is a short list of features: > *Monitor different instances of nTop (is the daemon running?) > *Automatically restart, and/or e-mail someone about, nTop instances that > go down unexpectedly. > *Persistently store data about each interface and each host, on an > hourly basis, and account for the counters being reset due to ntop being > restarted. > *The front-end displays most of the same data that the ntop embedded > front-end displays, except for data that ntop doesn't dump (such as port > distribution, per-host port usage, recent/active per-host sessions, > etc.). (In the future, I will write some scripts that extract certain > data from nTop's front-end, but that's a whole project in itself, so > it's pretty much dead last on my list.) > > Those are the main features - but the project is not complete, and I am > sure I will add more functionality before I begin using it regularly, > and before it is released publicly. > > I decided to write these scripts because I was asked to perform some > traffic and bandwidth utilization analysis for a mid-size company. nTop > is great software, but I needed to be able to permanently store its data > in a commonly used format, and most importantly, manage multiple > instances. My client has multiple sites spread across NYC, all of the > traffic goes to the central office, and and to their T3. They want > statistics for each site, individually. This script suite is designed > for this exact purpose - to manage multiple instances of nTop, > permanently store the data that it accumulates, and do this for a week > or two week long study. There will be such a large amount of data > accumulated throughout this study, that it was simply not reasonable to > have nTop dump its data to RRD tables, then manually graph different > data. This means I'd have to write a front-end that pulled data from > the RRD tables and displayed them, meaning it'd be easier, quicker, and > more space-friendly for me to have nTop dump the data, and have my > scripts pick and choose what to store permanently. > > I just wanted to give everyone a heads up, in case you might find this > at all useful. When this project is complete, and I start using these > scripts for the actual study, I will release what I have, hopefully > along with some decent documentation. > > Feel free to e-mail me with any questions. If anyone knows of an > appropriate location, associated with nTop, for me to post these > scripts, I'd be happy to do so. > > Thanks, and have a great day! > > -Jimmy Ryan > > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > <font size="1"> > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'> > </div> > "This email is intended to be reviewed by only the intended recipient > and may contain information that is privileged and/or confidential. > If you are not the intended recipient, you are hereby notified that > any review, use, dissemination, disclosure or copying of this email > and its attachments, if any, is strictly prohibited. If you have > received this email in error, please immediately notify the sender by > return email and delete this email from your system." > </font> > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop <font size="1"> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'> </div> "This email is intended to be reviewed by only the intended recipient and may contain information that is privileged and/or confidential. If you are not the intended recipient, you are hereby notified that any review, use, dissemination, disclosure or copying of this email and its attachments, if any, is strictly prohibited. If you have received this email in error, please immediately notify the sender by return email and delete this email from your system." </font> _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
