-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Marek Stopka wrote:
> A little bit of benchmarking... :-)
>
> livebox-flow:~/tmp$ time nfdump-1.6b-snapshot-20090619/bin/nfdump -f
> Supernetwork-nix-IN.filter.old -r nfcapd.200907141430 &>/dev/null
>
> real 0m16.960s
> user 0m12.585s
> sys 0m0.868s
>
>
> In this filter lot of ORs is used...
>
> livebox-flow:~/tmp$ time nfdump-1.6b-snapshot-20090619/bin/nfdump -f
> Supernetwork-nix-IN.filter -r nfcapd.200907141430 &>/dev/null
>
> real 0m11.090s
> user 0m3.604s
> sys 0m0.824s
>
> in this filter list syntax is used...
The benefit should be even bigger. As sys and real is not too different, you
may have some other potential bottlenecks
such as memory or I.O.
Also try to test the option to compress the flows. Depending on the relation of
CPU and I/O you can gain speed using
compression, as decompression is faster than transferring additional data in I/O
Another benchmark for the speedup of lists:
nfdump 1.6b 20090619
flowfile : 325MB
flows : 5.5 M
OR chain of 440 terms:
Total flows processed: 5411216, Blocks skipped: 0, Bytes read: 325280619
Sys: 47.950s flows/second: 112848.9 Wall: 48.285s flows/second: 112066.9
IP list
Total flows processed: 5411216, Blocks skipped: 0, Bytes read: 325280619
Sys: 1.840s flows/second: 2940696.1 Wall: 1.857s flows/second: 2913259.3
- Peter
>
> On Tue, Jul 14, 2009 at 8:46 PM, Peter Haag<peter.h...@switch.ch> wrote:
>
>
> Jan Pazdera wrote:
>>>> Hi all,
>>>>
> snipp ..
>>>>>
>>>> I would like to add my experience with filtering of huge data amounts. A
>>>> year ago, we used 32b nfsen/nfdump on dual core cpu and 4 GB RAM with
>>>> RAID5 to collect data with 75k flows per second on 10G network. Then we
>>>> applied a filter with approx. 2000 net masks in global OR (for Marek:
>>>> NIX traffic filter;). Nfprofile took about 8 minutes to finish its work
>>>> making the filter not appliable (as nfsen process data every 5 minutes),
>>>> we had to use sampling 1:10. But what was very strange: when I applied
>>>> the NOT on the whole filter, it took several times more time to finish
>>>> this filter. Why?
> Well - Everything is a balance between CPU and I/O. In your case in may
> be for sure the 2000 net masks in global OR. This kills filtering of large
> networks. That's why I introduced lists in nfdump filter syntax a while ago.
> Lists use red black search trees which are ways more effective for large lists
> as global OR chains.
>
> Try the syntax like 'src ip in [ 1.2.3.4/16 2.3.4.5/24 .. ] AND ...'
> This way - a couple of thousand expressions is very fast for filtering.
>
>>>> During this test it was not a problem with RAM which was not completely
>>>> used. Bottle neck was a CPU and especially the HDD.
>>>>
>>>> Next problem I faced was the old nfsen behaviour, when it ran nfprofile
>>>> sequently for every profile so only 1 cpu core was used. As far as I
>>>> know, this should be fixed in actual version, so extra memory and extra
>>>> cpu cores makes the better performance. Also use the fastest HDDs as you
>>>> can.
> Yes - this is a know problem. 1.3.2 does better use CPU power for profiling.
> I hope to improve that even more, as the current fix is a simple but working
> work around the "bug"
>
>>>> But what is important: It is also not very important how much traffic
>>>> you meassure. The main factor slowing the netflow data processing is the
>>>> number of flows. Try to decrease their number by increasing the flow
>>>> cache on your netflow generators and increasing the active and inactive
>>>> timeouts.
>>>>
>>>> As far as I know, NfSen is the only tool able to process netflow from
>>>> saturated 10GB traffic with more than 40k flows per second. Good work,
>>>> Peter!
> Thanks again!
>
> - Peter
>
>>>> Regards,
>>>> HOP
>>>>>
>>>>> Please note, if you really process high volume flow files, you should
>>>>> consider to migrate to a 64bit system. nfdump
>>>>> takes as much memory as it requires, which is the most influencing
>>>>> factor for speed. Memory can not be replaced except
>>>>> by memory! having enough memory an I.O capacity is the credo.
>>>>>
>>>>> Hope this helps
>>>>>
>>>>> - Peter
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ------------------------------------------------------------------------------
>>>>
>>>> Enter the BlackBerry Developer Challenge This is your chance to win up
>>>> to $100,000 in prizes! For a limited time,
>>>> vendors submitting new applications to BlackBerry App World(TM) will have
>>>> the opportunity to enter the BlackBerry Developer Challenge. See full
>>>> prize details at: http://p.sf.net/sfu/Challenge
>>>> _______________________________________________
>>>> Nfdump-discuss mailing list
>>>> Nfdump-discuss@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
>>>>
>>
- --
_______ SWITCH - The Swiss Education and Research Network ______
Peter Haag, Security Engineer, Member of SWITCH CERT
PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7
SWITCH, Werdstrasse 2, P.O. Box, CH-8021 Zurich, Switzerland
E-mail: peter.h...@switch.ch Web: http://www.switch.ch/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
iQCVAwUBSl2LAv5AbZRALNr/AQJ0VQP8DxBKKJe/+uqBfnzULNWQC01EHc47+Cuh
lXEAHSPi/V/e0Rqtl0ojXu1Z1fBs2AdcF6V3WrMCQP/sXOXLX6tu4lf74pWLbOTS
itRd1pm3whM0Qod9ew291pLSz09AWSQaA+YTfyMI33BClYwOfqvrdxaABL9sEC2U
lbzVEw37CvM=
=sZR/
-----END PGP SIGNATURE-----
------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Nfdump-discuss mailing list
Nfdump-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
