-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Paul,
- --On March 1, 2007 15:41:12 +0000 Paul Mullen <[EMAIL PROTECTED]> wrote:
| Hi,
|
| We have a fairly large setup for nfsen here and as part of the reports we run
| daily are finding top uploader and downloader subnets and some other stats.
| There are ~4000 address blocks as well as ~4000 link addresses to these
| blocks and as such we are running nfdump about 16000 times to generate these
| reports. This is taking about 22 hours.
Not sure, if I understand you right. Does it mean you want to have the flows
for each IP address in a separate file? Or how would a sample filter look like?
|
| I plan on hacking the nfdump source to allow it to run all filters on the
| files in a single pass. Am I looking a 2 ways of doing it. 1) Brute force:
| Create a Filter_Engine for each filter and run each one on each record. Not
| sure if this will gain me much in the way of performance. 2) Modify
That's exactly the way nfprofile works. You have a Filter_Engine for each
channel.
As disk IO is a major issue, this speeds up the process quite a lot. But I'm not
sure if this still works for 16000 filters, as the number of open files per
process
will limit you.
| Filter_Engine: Modify the filter engine to have load different filters and to
| store the filter number in onTrue so that it is returned to the record
| processor.
If I understand you right, you will have one large filter with 16k IP addresses
and you want to know which one matched? If so some, the latest nfdump snapshot
could help you here, as it implements IP lists for a very fast IP matching of
about
several thousand of different IPs. When found, the list can tell you exactly
which
one it was. At the moment the information is discarded, as not needed.
- Peter
|
| Most likely there is a vastly superior 3rd option but I'm not sure what it is.
| Any guidance would be much appreciated.
|
| Paul.
- --
_______ SWITCH - The Swiss Education and Research Network ______
Peter Haag, Security Engineer, Member of SWITCH CERT
PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7
SWITCH, Limmatquai 138, CH-8001 Zurich, Switzerland
E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)
iQCVAwUBReb84P5AbZRALNr/AQJiawP+Klh/8m6YsG6aECciTCafyRmVaJfpX+Sa
GOTPkoX/JztT/fLki2u1wuWqg3mC/wMuTzBUH4G2mx6ULVOeLAMn2m1seHDN7t+L
4+gwnBvAHuP2KBHNjEdjxfkGMjoJxWykiKmmArLn/F+F+Axc00LoNOqDj2l4HzKq
BMrE74yd7PA=
=8FOx
-----END PGP SIGNATURE-----
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nfdump-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
