Hi all,
Peter Haag napsal(a):
> Hi Marek,
>
>
>
> The speed of the filter engine depends on various things, and a
> general answer is rather complex. The filter tries to
> optimise as good as possible and evaluates only required expressions
> and skips non relevant terms. you need many hundred
> or thousand of terms until a noticeable speed penalty occurs, as other
> system resources are more relevant for speed,
> such as i.O.
>
> The worst filter term is a single chain of .. OR .. OR .. OR .. . If
> this reaches many hundred terms, the list syntax is
> way more efficient:
> .. in [ list ]
>
> So, there is no general answer, however, the filter engine is seldom
> the bottleneck. I have very large filters and
> nfdump performs well.
>
>
I would like to add my experience with filtering of huge data amounts. A
year ago, we used 32b nfsen/nfdump on dual core cpu and 4 GB RAM with
RAID5 to collect data with 75k flows per second on 10G network. Then we
applied a filter with approx. 2000 net masks in global OR (for Marek:
NIX traffic filter;). Nfprofile took about 8 minutes to finish its work
making the filter not appliable (as nfsen process data every 5 minutes),
we had to use sampling 1:10. But what was very strange: when I applied
the NOT on the whole filter, it took several times more time to finish
this filter. Why?
During this test it was not a problem with RAM which was not completely
used. Bottle neck was a CPU and especially the HDD.
Next problem I faced was the old nfsen behaviour, when it ran nfprofile
sequently for every profile so only 1 cpu core was used. As far as I
know, this should be fixed in actual version, so extra memory and extra
cpu cores makes the better performance. Also use the fastest HDDs as you
can.
But what is important: It is also not very important how much traffic
you meassure. The main factor slowing the netflow data processing is the
number of flows. Try to decrease their number by increasing the flow
cache on your netflow generators and increasing the active and inactive
timeouts.
As far as I know, NfSen is the only tool able to process netflow from
saturated 10GB traffic with more than 40k flows per second. Good work,
Peter!
Regards,
HOP
>
>
> Please note, if you really process high volume flow files, you should
> consider to migrate to a 64bit system. nfdump
> takes as much memory as it requires, which is the most influencing
> factor for speed. Memory can not be replaced except
> by memory! having enough memory an I.O capacity is the credo.
>
> Hope this helps
>
> - Peter
>
>
>
>
>
------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full
prize
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Nfdump-discuss mailing list
Nfdump-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
--
Jan Pazdera <pazdera at invea.cz>
INVEA-TECH a.s.
U Vodarny 2965/2, 61600 Brno, Czech Republic
Tel: +420 541 142 051
www.invea-tech.com
Key 0x89F62F78: 41A7 28C2 C624 FBD1 E236 6827 42EB 3694 89F6 2F78
------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Nfdump-discuss mailing list
Nfdump-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
