Peter Haag wrote: > Hi Maurizio, > > --On January 16, 2007 18:48:02 +0000 Maurizio Molina > <[EMAIL PROTECTED]> wrote: > > | Hi Peter, > | I installed the latst nfdump snapshot and your suggestion worked, except > | that I had to give an explicit -n 1000 (the maximum allowed), because > | with -n 0 it complains: > | [EMAIL PROTECTED] bin]$ ./nfdump -M > | > /opt/flowtest/tools/nfdump/snapshot-20070110/archive/ath_gr:bud_hu:lon_uk > | -R nfcapd.200701161750:nfcapd.200701161750 -s record/bytes -A > | srcas,dstas -n 0 -o "fmt:%sas %das %byt %pkt %fl" > | TopN for record statistic: 0 < topN < 1000 only allowed for IP > statistics > > As the flow stat routines are optimized for speed, two different > sorting routines > are implemented. The flow sorter is very fast <100 and acceptable for > <1000 but gets slow > for n > 1000. If you think, speed is not an issue for you you may > remove this limit by > removing the test at nfdump.c line 1191 or setting 1000 to 10000. > Never set topN to 0.
OK, I did it. I understand the motivation for puttig this limit, but for some applications (e.g. the AS AS matrix generation I have) it is a limitation. Why don't you remove this limit from nfdump (so that users directly using nfdump are not constrained) and keep this limit only in nfsen, as it is now, where you currently can ask for 10, 20, 50, 100, 500 and 1000 flows? Regards, Maurzio > > - Peter > | > | However, -n 1000 is a limitation, for me, since I have more than 35 > | peering AS, so the src_as->dst_as matrix has potentially > 1000 > | elements (even if not all the AS have a traffic relationship, so some > | elements may be absent...but I would like not to count on that...) > | Regards, > | Maurizio > | > | > | Peter Haag wrote: > | > | > Hi Maurizio, > | > > | > An AS-AS matrix can be created more easily as follows: > | > > | > ./nfdump -M <source_list> -R nfcapd.$tart_tslot:nfcapd.$end_tslot > | > -s record/bytes -A srcas,dstas -n 0 -o "fmt:%sas %das %byt" > | > > | > This generates you a list of all AS to AS relations, with a custom > | > output format. You may of course add any additional field in the > | > custom output format, you may need for your purpose. This output can > | > be easily parsed and used for further processing. > | > > | > Therefore a single run gives all required information, no need for > | > filtering either, and therefore no need for parallel filters, which > | > btw. is the way nfprofile handles multiple channels :) > | > > | > Hope this helps > | > > | > - Peter > | > > | > -------- Original Message -------- > | > From: Maurizio Molina <[EMAIL PROTECTED]> > | > To: nfsen-discuss ML <[email protected]> > | > Subject: [Nfsen-discuss] AS-AS traffic matric - backend plugin > | > Date: Tue Oct 24 2006 18:05:16 GMT+0200 (CEST) > | > > | > > Hi, > | > > I'm writing a backend plugin to obtain a daily AS-AS traffic > matric in > | > > my network, with 38 ASs and 21 sources. > | > > The only way I found so far is to get the information with > nfdump (1.5) > | > > running > | > > | > ># nfdump -M <source_list> -R > nfcapd.$tart_tslot:nfcapd.$end_tslot -n 50 > | > > -s srcas/bytes -o long "src as $src_as and dst as $dst_as" > | > > | > > as many times as all the possible AS-AS pairs (38X38), and then > parse > | > > the output. > | > > Note that I use -n 50 but I could vell have used -n 1 (because > of the > | > > filtering, I always get that there is only one contributing > src_as). > | > > The problem is that given the number of flows (roughly: 300 k > flows per > | > > source and per hour, with each AS connected to one, or two, or three > | > > sources at most), the processing time is high. > | > > I probably won't be able to run the processing every day over > all the > | > > past 24 hours, but I'll be forced to focus on a limited time slice. > | > > Questions: > | > > 1) is there another easy way to do? > | > > 2) if not, how difficult would it be (and what module should be > | > > modified) to let nfdump have prallel filters? The processing > bottleneck > | > > is clearly the disk access bandwidth (the cpu stays at about 4-5%). > | > > | > > Regards, > | > > Maurizio > | > > | > > | > > | > > > ------------------------------------------------------------------------- > | > > Using Tomcat but need to do more? Need to support web services, > security? > | > > Get stuff done quickly with pre-integrated technology to make your > | > job easier > | > > Download IBM WebSphere Application Server v.1.0.1 based on Apache > | > Geronimo > | > > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > | > > _______________________________________________ > | > > Nfsen-discuss mailing list > | > > [email protected] > | > > https://lists.sourceforge.net/lists/listinfo/nfsen-discuss > | > > | > > | > -- > | > _______ SWITCH - The Swiss Education and Research Network ______ > | > Peter Haag, Security Engineer, Member of SWITCH CERT > | > PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7 > | > SWITCH, Limmatquai 138, CH-8001 Zurich, Switzerland > | > E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/security > | > | > | > | > | > ------------------------------------------------------------------------- > | Take Surveys. Earn Cash. Influence the Future of IT > | Join SourceForge.net's Techsay panel and you'll get the chance to > share your > | opinions on IT & business topics through brief surveys - and earn cash > | > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > | _______________________________________________ > | Nfsen-discuss mailing list > | [email protected] > | https://lists.sourceforge.net/lists/listinfo/nfsen-discuss > > > -- ______________________________________________________________________ Maurizio Molina Network Engineer DANTE - www.dante.net Tel: +44 (0)1223 371 300 Fax: +44 (0)1223 371 371 Email: [EMAIL PROTECTED] PGP Key ID: 3FF58D51 City House, 126-130 Hills Road Cambridge CB2 1PQ UK _____________________________________________________________________ ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Nfsen-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
