Hello, A little of context: I'm doing a simple network flow analysis based on a few days of traffic. The overall duration of the collecting process will be no more than 5 to 7 days.
Because this is a limited time experience, I think I prefer to do a post processing of tcpdump files, that way avoiding to do some filtering/optimization to what traffic gets processed, as it would not reflect 100% the traffic on the network. For continued analysis this method would be impossible to achieve and, in my opinion, kind of stupid. More memory, would be the best solution of course :) But not feasible in the few days I have left. I already collected a day or so of data, by Sunday I will have all data collected. I will then Zip the files and upload them to the 1GB ram machine. The uploading will be done after all data is collected, thus not affecting bandwidth usage. A couple of days ago, I read a few things about nProbe that caught my attention, but because it isn't free, I chose to use Ntop instead. Thanks Burton! -pfeito > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Burton > M. Strauss III > Sent: quarta-feira, 28 de Abril de 2004 14:43 > To: [EMAIL PROTECTED] > Subject: RE: [Ntop] Post processing of tcpdump files with NTOP > > Three choices... > > 1. Use the -x and/or -X options to limit the number of hosts ntop > processes > to what fits in memory. Crude, but maybe workable. > > 2. Use filtering to limit it to the important hosts and/or some of the > workload reduction options - man ntop. Better than #1, but takes more > knowledge of your environment. > > 3. Buy more memory. 512MB DDR is still under US$70 if you shop carefully. > Two Saturday's ago BestBuy ad, for example, PC2700 DDR 512MB US$90 less > US$35 Mail-in-rebate. > > 4. Use a netFlow collector (nProbe, etc.) on the local machine sending the > data to the remote. > > 5. Use the capture files - but isn't the transport of them causing more > bandwidth usage??? > > OK, that's 5, but ... you get the drift. > > > -----Burton > > > > -----Original Message----- > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of > > pfeito > > Sent: Wednesday, April 28, 2004 7:54 AM > > To: [EMAIL PROTECTED] > > Subject: RE: [Ntop] Post processing of tcpdump files with NTOP > > > > > > > > Hello again, > > > > > -----Original Message----- > > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On > > Behalf Of Burton > > > M. Strauss III > > > Sent: quarta-feira, 28 de Abril de 2004 4:21 > > > To: [EMAIL PROTECTED] > > > Subject: RE: [Ntop] Post processing of tcpdump files with NTOP > > > > > > Which version of ntop? 3.0 is MUCH more stable than the 2.2 > > series. And > > > should have no problems. > > > > Im using 3.0. > > > > > > > > You are wrong about memory usage - ntop doesn't benefit from running > > > off-line, in fact will probably need more memory because it > > won't be able > > > to > > > purge inactive hosts. > > > > I believe so, but in this specific case, the data collecting PC has only > > 256MB Ram which becomes exausted after +- 10H when using NTOP in real- > time > > processing mode (due to the large amounts of traffic in the network). > > > > I also have a remote machine with 1GB ram which I can use, but > > not connected > > to the target network, therefore I can only use it to process previous > > collected data. It takes more memory, but with 1GB I could > > process more than > > than 10H. > > > > Ideally, it would be better if I could deploy the 1GB ram machine in the > > target network and use NTOP in real-time, but this is not the case, > > unfortunely. > > > > > > > > Certainly the data you're looking for is in the rrd files - you may > need > > > to > > > create some custom graphs using rrdtool, but the data is there. > > > > That is interesting. I've to research further on that. I don't have a > clue > > how can I make custom graphs with rrdtool, since I dont know much about > > rrdtool, only that is used to collect periodically. > > > > Thanks for the feedback :) > > -pfeito > > > > > > > > -----Burton > > > > > > > > > > -----Original Message----- > > > > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Behalf Of > > > > pfeito > > > > Sent: Tuesday, April 27, 2004 7:28 PM > > > > To: [EMAIL PROTECTED] > > > > Subject: [Ntop] Post processing of tcpdump files with NTOP > > > > > > > > > > > > Hi to all! > > > > > > > > > > > > > > > > I'm trying to do some network analysis in a university department > > > network > > > > and I choose to use NTOP to acquire statistic data. 10 hours > > later NTOP > > > > crashed due to lack of memory (only 256MB were available in the NTOP > > > > machine). > > > > > > > > > > > > > > > > I spent a 2 or 3 hours reading some references, trying to > > understand the > > > > memory limitations of NTOP, and, if I understood well, its is kind > of > > > > difficult to do a long run analysis (e.g. 1 week or +) with NTOP > when > > > > dealing with medium size to large networks, although it really > depends > > > on > > > > the machine specs. > > > > > > > > > > > > > > > > I decided to try a different approach: to collect raw tcpdump > > output for > > > a > > > > week, and then feed that data to NTOP. I've done a little > > > > experiment with an > > > > 1 minute tcpdump file and it seem to work well. > > > > > > > > > > > > > > > > Will this method work for 1 week tcpdump file ? I suspect that the > > > memory > > > > limitation still poses a problem, but I could do post processing in > > > > different machine (i.e. with 1GB Ram). It seems to me that > > this offline > > > > processing method should need less memory compared with real-time > > > > processing > > > > mode. > > > > > > > > > > > > > > > > Any feedback from people that has actually done some data processing > > > like > > > > this would be appreciated :) > > > > > > > > > > > > > > > > BTW: as I saw in another post, the tcpdump file only worked when one > > > > specific interface is indicated with -i parameter (e.g. > > tcpdump -i eth0 > > > -w > > > > dumpfile) > > > > > > > > > > > > > > > > -pfeito > > > > > > > > > > > > > > _______________________________________________ > > > Ntop mailing list > > > [EMAIL PROTECTED] > > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > _______________________________________________ > > Ntop mailing list > > [EMAIL PROTECTED] > > http://listgateway.unipi.it/mailman/listinfo/ntop > > _______________________________________________ > Ntop mailing list > [EMAIL PROTECTED] > http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [EMAIL PROTECTED] http://listgateway.unipi.it/mailman/listinfo/ntop
