Hi Peter, On Thursday 01 March 2007, Peter Haag wrote: > Hi Paul, > > --On March 1, 2007 15:41:12 +0000 Paul Mullen <[EMAIL PROTECTED]> wrote: > | Hi, > | > | We have a fairly large setup for nfsen here and as part of the reports we > | run daily are finding top uploader and downloader subnets and some other > | stats. There are ~4000 address blocks as well as ~4000 link addresses to > | these blocks and as such we are running nfdump about 16000 times to > | generate these reports. This is taking about 22 hours. > > Not sure, if I understand you right. Does it mean you want to have the > flows for each IP address in a separate file? Or how would a sample filter > look like? The way it is currently set up is that we have a number of end user sites ~4000 that connect through 7 different service providers and each of these providers gives us a tunnel to an aggregation router. We currently have a directory per aggregation router. The filter we run for each end site looks like
nfdump -q -m -Rnfcapd.200702280000:nfcapd.200702282355 -a -A srcip4/<netmask> "(SRC NET <network/netmask>)" nfdump -q -m -Rnfcapd.200702280000:nfcapd.200702282355 -a -A destip4/<netmask> "(DEST NET <network/netmask>)" > > | I plan on hacking the nfdump source to allow it to run all filters on the > | files in a single pass. Am I looking a 2 ways of doing it. 1) Brute > | force: Create a Filter_Engine for each filter and run each one on each > | record. Not sure if this will gain me much in the way of performance. 2) > | Modify > > That's exactly the way nfprofile works. You have a Filter_Engine for each > channel. As disk IO is a major issue, this speeds up the process quite a > lot. But I'm not sure if this still works for 16000 filters, as the number > of open files per process will limit you. We do profiles per service provider at the minute and it might make sense to run all these checks on the profiles instead of the live profile. > > | Filter_Engine: Modify the filter engine to have load different filters > | and to store the filter number in onTrue so that it is returned to the > | record processor. > > If I understand you right, you will have one large filter with 16k IP > addresses and you want to know which one matched? If so some, the latest > nfdump snapshot could help you here, as it implements IP lists for a very > fast IP matching of about several thousand of different IPs. When found, > the list can tell you exactly which one it was. At the moment the > information is discarded, as not needed. This sounds extremely useful and I'll look into it. Paul. > > - Peter > > | Most likely there is a vastly superior 3rd option but I'm not sure what > | it is. Any guidance would be much appreciated. > | > | Paul.
pgpC1dfznD3OR.pgp
Description: PGP signature
------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________ Nfdump-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
