Thank you for your reply, I'd like to monitor disk usage, but I don't have iostat installed (i'm running Red Hat enterprise linux 4), but I will google for an alternative.
I don't know yet how to use channels (and I'd probably wait for some public documentation), but I'll see what I can do. Thanks again. Peter Haag wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi Adrian, > > - --On February 1, 2007 15:28:32 +0200 Adrian Popa <[EMAIL PROTECTED]> wrote: > > | Hello, > | > | I have a question about the performance of nfdump, but first, let me > | explain what I'm trying to do: > | I have a plugin that searches the collected flows for specific network > | prefixes (or AS-es) on each exporting router, on specific intefaces. The > | information is then fed into custom rrd files and plotted as png images. > | Searching is done by using a top 1 record/bytes and filtering by 'inif x > | and net 1.2.3.0/24'. Here's an example: > | > | $nfdump -r /data/nfsen/profiles/live/$border/nfcapd.$timeslot -n 1 -s > | record/bytes -o "fmt:%ts %td %pr %sap -> %dap %pkt %byt %bps %in %out > | %sas %das %fl" '$ifType $if and net $prefix' > | > | I have to search for input traffic on a specific interface for a > | specific network prefix and also for output traffic for the same thing. > | > | I achieved to do this, and it works well, but execution time for 2 > | borders (with 3-4 interfaces each), 20 prefixes and 50 AS-es for a peak > | traffic of about 2Gbps takes about 3,5 minutes. > > If I understand you right, you are going to call this nfdump command for > each of the prefixes, which results in a lot of sequential nfdump commands. > > | > | In the future I will want to monitor other routers, on the same > | principle. As far as I see, I can do that, but either I gather less > | data, or I use a different machine for collecting. > | > | A colleague of mine proposed that I split my script (which is 100% > | sequential) into several threads that run at the same time. Each thread > | would call nfdump and update it's particular rrd. > | > | My question to you is this: Assuming that I start the new processes like > | threads (or more likely like forked processes), would I get a speed > | increase? I'd like to say that this script keeps the processor usage at > | about 60-80%. > > The CPU usage is only half of the story for your plugin. Almost every time > I/O is much more a problem. For each nfdump command you read a lot of data. > If this amount of data does not fit into the file system cache of your OS, > the performance rapidly drops. So I'd recommend you to analyse how your > system behaves in such a plugin cycle. Have a look at the IO using iostat > check the service time of your disks. If you still have room creating > more threads can result in better performance. If your IO system is at its > limit, you will not gain anything, and CPU will stay at 60-80% as your system > has lots of IO wait. Overcoming this, you would need more RAM to increase the > available memory for the IO file system cache. A high service time in IO stat > means slow disks - so you'll need the right balance of disks and memory. > > Furthermore you can try to optimise IO by limiting reading data once and doing > parallel processing - the way which nfprofile does profiling all your > channels. > It reads the data once only and applies all filters to the same data. > > Coming back to your plugin - you may try to optimise IO by creating a profile > with a channel per prefix and creating adequate filters per channel. Your > plugin then needs to create the stat Top 1 per channel which may result in > reading > less data over all - but this is just a guess. > > So - it's a bit of all. > Hope this helps anyway. > > - Peter > > | > | I don't know if the forked processes would load the same input file into > | memory again and again, or if they would share the same file (lowering > | memory consumption)? > | > | What are your recomandations? > | > | Thank you for your time, > | > | -- > | Adrian Popa > | > | > | > | ------------------------------------------------------------------------- > | Using Tomcat but need to do more? Need to support web services, security? > | Get stuff done quickly with pre-integrated technology to make your job > easier. > | Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > | http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > | _______________________________________________ > | Nfsen-discuss mailing list > | [email protected] > | https://lists.sourceforge.net/lists/listinfo/nfsen-discuss > > > > - -- > _______ SWITCH - The Swiss Education and Research Network ______ > Peter Haag, Security Engineer, Member of SWITCH CERT > PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7 > SWITCH, Limmatquai 138, CH-8001 Zurich, Switzerland > E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.3 (Darwin) > > iQCVAwUBRcMAaP5AbZRALNr/AQLXEwP/bymvl/R3I5MqF8qXSq82QXwDng9VPcyH > 56KfUdgDFYpVSOM/Jjn08t8LPaGCA/2DQFxzjzXc+g/YngLfOFZFxkjZEDfRo3AS > 53T8cZeTHIx8gy4Xn1y5VqerK0+Q4BNB+I+1+YYo/g8wVfE+pBMNNKh1m+krIwLO > isZnG514jMA= > =E3HQ > -----END PGP SIGNATURE----- > > > -- Adrian Popa Junior Network Engineer Romtelecom S.A. Divizia Centrul National de Operare Retea Departament Transport IP & Metro Compartiment IP Core ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Nfsen-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
