--On January 17, 2007 7:25:06 PM +0000 Maurizio Molina <[EMAIL PROTECTED]>
wrote:
| Peter Haag wrote:
|
| > Hi Maurizio,
| >
| > --On January 16, 2007 18:48:02 +0000 Maurizio Molina
| > <[EMAIL PROTECTED]> wrote:
| >
| > | Hi Peter,
| > | I installed the latst nfdump snapshot and your suggestion worked, except
| > | that I had to give an explicit -n 1000 (the maximum allowed), because
| > | with -n 0 it complains:
| > | [EMAIL PROTECTED] bin]$ ./nfdump -M
| > |
| > /opt/flowtest/tools/nfdump/snapshot-20070110/archive/ath_gr:bud_hu:lon_uk
| > | -R nfcapd.200701161750:nfcapd.200701161750 -s record/bytes -A
| > | srcas,dstas -n 0 -o "fmt:%sas %das %byt %pkt %fl"
| > | TopN for record statistic: 0 < topN < 1000 only allowed for IP
| > statistics
| >
| > As the flow stat routines are optimized for speed, two different
| > sorting routines
| > are implemented. The flow sorter is very fast <100 and acceptable for
| > <1000 but gets slow
| > for n > 1000. If you think, speed is not an issue for you you may
| > remove this limit by
| > removing the test at nfdump.c line 1191 or setting 1000 to 10000.
| > Never set topN to 0.
|
| OK, I did it.
| I understand the motivation for puttig this limit, but for some
| applications (e.g. the AS AS matrix generation I have) it is a limitation.
| Why don't you remove this limit from nfdump (so that users directly
| using nfdump are not constrained) and keep this limit only in nfsen, as
| it is now, where you currently can ask for 10, 20, 50, 100, 500 and
| 1000 flows?
I will change that, with a transparent replace of the sorter algorithm for n >
1000
It's on the todo list.
- Peter
|
| Regards,
| Maurzio
|
| >
| > - Peter
| > |
| > | However, -n 1000 is a limitation, for me, since I have more than 35
| > | peering AS, so the src_as->dst_as matrix has potentially > 1000
| > | elements (even if not all the AS have a traffic relationship, so some
| > | elements may be absent...but I would like not to count on that...)
| > | Regards,
| > | Maurizio
| > |
| > |
| > | Peter Haag wrote:
| > |
| > | > Hi Maurizio,
| > | >
| > | > An AS-AS matrix can be created more easily as follows:
| > | >
| > | > ./nfdump -M <source_list> -R nfcapd.$tart_tslot:nfcapd.$end_tslot
| > | > -s record/bytes -A srcas,dstas -n 0 -o "fmt:%sas %das %byt"
| > | >
| > | > This generates you a list of all AS to AS relations, with a custom
| > | > output format. You may of course add any additional field in the
| > | > custom output format, you may need for your purpose. This output can
| > | > be easily parsed and used for further processing.
| > | >
| > | > Therefore a single run gives all required information, no need for
| > | > filtering either, and therefore no need for parallel filters, which
| > | > btw. is the way nfprofile handles multiple channels :)
| > | >
| > | > Hope this helps
| > | >
| > | > - Peter
| > | >
| > | > -------- Original Message --------
| > | > From: Maurizio Molina <[EMAIL PROTECTED]>
| > | > To: nfsen-discuss ML <[email protected]>
| > | > Subject: [Nfsen-discuss] AS-AS traffic matric - backend plugin
| > | > Date: Tue Oct 24 2006 18:05:16 GMT+0200 (CEST)
| > | >
| > | > > Hi,
| > | > > I'm writing a backend plugin to obtain a daily AS-AS traffic
| > matric in
| > | > > my network, with 38 ASs and 21 sources.
| > | > > The only way I found so far is to get the information with
| > nfdump (1.5)
| > | > > running
| > | >
| > | > ># nfdump -M <source_list> -R
| > nfcapd.$tart_tslot:nfcapd.$end_tslot -n 50
| > | > > -s srcas/bytes -o long "src as $src_as and dst as $dst_as"
| > | >
| > | > > as many times as all the possible AS-AS pairs (38X38), and then
| > parse
| > | > > the output.
| > | > > Note that I use -n 50 but I could vell have used -n 1 (because
| > of the
| > | > > filtering, I always get that there is only one contributing
| > src_as).
| > | > > The problem is that given the number of flows (roughly: 300 k
| > flows per
| > | > > source and per hour, with each AS connected to one, or two, or three
| > | > > sources at most), the processing time is high.
| > | > > I probably won't be able to run the processing every day over
| > all the
| > | > > past 24 hours, but I'll be forced to focus on a limited time slice.
| > | > > Questions:
| > | > > 1) is there another easy way to do?
| > | > > 2) if not, how difficult would it be (and what module should be
| > | > > modified) to let nfdump have prallel filters? The processing
| > bottleneck
| > | > > is clearly the disk access bandwidth (the cpu stays at about 4-5%).
| > | >
| > | > > Regards,
| > | > > Maurizio
| > | >
| > | >
| > | >
| > | > >
| > -------------------------------------------------------------------------
| > | > > Using Tomcat but need to do more? Need to support web services,
| > security?
| > | > > Get stuff done quickly with pre-integrated technology to make your
| > | > job easier
| > | > > Download IBM WebSphere Application Server v.1.0.1 based on Apache
| > | > Geronimo
| > | > >
| > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
| > | > > _______________________________________________
| > | > > Nfsen-discuss mailing list
| > | > > [email protected]
| > | > > https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
| > | >
| > | >
| > | > --
| > | > _______ SWITCH - The Swiss Education and Research Network ______
| > | > Peter Haag, Security Engineer, Member of SWITCH CERT
| > | > PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7
| > | > SWITCH, Limmatquai 138, CH-8001 Zurich, Switzerland
| > | > E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/security
| > |
| > |
| > |
| > |
| > |
| > -------------------------------------------------------------------------
| > | Take Surveys. Earn Cash. Influence the Future of IT
| > | Join SourceForge.net's Techsay panel and you'll get the chance to
| > share your
| > | opinions on IT & business topics through brief surveys - and earn cash
| > |
| > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
| > | _______________________________________________
| > | Nfsen-discuss mailing list
| > | [email protected]
| > | https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
| >
| >
| >
|
| --
| ______________________________________________________________________
|
| Maurizio Molina
| Network Engineer
|
| DANTE - www.dante.net
|
| Tel: +44 (0)1223 371 300
| Fax: +44 (0)1223 371 371
| Email: [EMAIL PROTECTED]
| PGP Key ID: 3FF58D51
|
| City House, 126-130 Hills Road
| Cambridge CB2 1PQ
| UK
| _____________________________________________________________________
|
--
_______ SWITCH - The Swiss Education and Research Network ______
Peter Haag, Security Engineer, Member of SWITCH CERT
PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7
SWITCH, Limmatquai 138, CH-8001 Zurich, Switzerland
E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys - and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nfsen-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
