-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Adrian,
- --On February 1, 2007 15:28:32 +0200 Adrian Popa <[EMAIL PROTECTED]> wrote:
| Hello,
|
| I have a question about the performance of nfdump, but first, let me
| explain what I'm trying to do:
| I have a plugin that searches the collected flows for specific network
| prefixes (or AS-es) on each exporting router, on specific intefaces. The
| information is then fed into custom rrd files and plotted as png images.
| Searching is done by using a top 1 record/bytes and filtering by 'inif x
| and net 1.2.3.0/24'. Here's an example:
|
| $nfdump -r /data/nfsen/profiles/live/$border/nfcapd.$timeslot -n 1 -s
| record/bytes -o "fmt:%ts %td %pr %sap -> %dap %pkt %byt %bps %in %out
| %sas %das %fl" '$ifType $if and net $prefix'
|
| I have to search for input traffic on a specific interface for a
| specific network prefix and also for output traffic for the same thing.
|
| I achieved to do this, and it works well, but execution time for 2
| borders (with 3-4 interfaces each), 20 prefixes and 50 AS-es for a peak
| traffic of about 2Gbps takes about 3,5 minutes.
If I understand you right, you are going to call this nfdump command for
each of the prefixes, which results in a lot of sequential nfdump commands.
|
| In the future I will want to monitor other routers, on the same
| principle. As far as I see, I can do that, but either I gather less
| data, or I use a different machine for collecting.
|
| A colleague of mine proposed that I split my script (which is 100%
| sequential) into several threads that run at the same time. Each thread
| would call nfdump and update it's particular rrd.
|
| My question to you is this: Assuming that I start the new processes like
| threads (or more likely like forked processes), would I get a speed
| increase? I'd like to say that this script keeps the processor usage at
| about 60-80%.
The CPU usage is only half of the story for your plugin. Almost every time
I/O is much more a problem. For each nfdump command you read a lot of data.
If this amount of data does not fit into the file system cache of your OS,
the performance rapidly drops. So I'd recommend you to analyse how your
system behaves in such a plugin cycle. Have a look at the IO using iostat
check the service time of your disks. If you still have room creating
more threads can result in better performance. If your IO system is at its
limit, you will not gain anything, and CPU will stay at 60-80% as your system
has lots of IO wait. Overcoming this, you would need more RAM to increase the
available memory for the IO file system cache. A high service time in IO stat
means slow disks - so you'll need the right balance of disks and memory.
Furthermore you can try to optimise IO by limiting reading data once and doing
parallel processing - the way which nfprofile does profiling all your channels.
It reads the data once only and applies all filters to the same data.
Coming back to your plugin - you may try to optimise IO by creating a profile
with a channel per prefix and creating adequate filters per channel. Your
plugin then needs to create the stat Top 1 per channel which may result in
reading
less data over all - but this is just a guess.
So - it's a bit of all.
Hope this helps anyway.
- Peter
|
| I don't know if the forked processes would load the same input file into
| memory again and again, or if they would share the same file (lowering
| memory consumption)?
|
| What are your recomandations?
|
| Thank you for your time,
|
| --
| Adrian Popa
|
|
|
| -------------------------------------------------------------------------
| Using Tomcat but need to do more? Need to support web services, security?
| Get stuff done quickly with pre-integrated technology to make your job easier.
| Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
| http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
| _______________________________________________
| Nfsen-discuss mailing list
| [email protected]
| https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
- --
_______ SWITCH - The Swiss Education and Research Network ______
Peter Haag, Security Engineer, Member of SWITCH CERT
PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7
SWITCH, Limmatquai 138, CH-8001 Zurich, Switzerland
E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)
iQCVAwUBRcMAaP5AbZRALNr/AQLXEwP/bymvl/R3I5MqF8qXSq82QXwDng9VPcyH
56KfUdgDFYpVSOM/Jjn08t8LPaGCA/2DQFxzjzXc+g/YngLfOFZFxkjZEDfRo3AS
53T8cZeTHIx8gy4Xn1y5VqerK0+Q4BNB+I+1+YYo/g8wVfE+pBMNNKh1m+krIwLO
isZnG514jMA=
=E3HQ
-----END PGP SIGNATURE-----
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Nfsen-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
