Peter, thank you for your prompt reply. Your information will help me make some tests and then decide if what I want is manageable...
If I undestood correctly, nfcapd collects data for 5 minutes, and then passes it to nfsen (and nfsen distributes it to different profiles/channels). This means that the filters are applied 'offline', and applying the filters to all profiles will take roughly the same amount of time it takes now with my plug-in. I had hoped that nfcapd was able to apply the filters when collecting data (it had to do this in real time) for me to 'win' computational time. This way, after 5 minutes of data collecting, I would have updated graphs for 200 profiles without additional computation. What you say is that after 5 minutes, data is filtered into the existing profiles and this filtering and writing to disk takes a while. For me, it means that the time may exceed the 5 minute deadline. The additional problem with this is that the plugins are run after the profiles are populated (or so I think). This means that my floodsearch plugin may run 4 minutes after the live profile has its data - which is not what I want. I believe that the 'features' I want imply significantly modifying nfcapd - which may not be possible, because in your design, nfcapd must be fast enough not to drop incoming packets (which, after all is more important!)... I fear that the only sustainable solution would be to separate the work load on other collectors. To add some numbers to this, here's what I have now: System: IBM, 2 x Intel(R) Xeon(TM) CPU 3.40GHz, 7GB RAM, RAID 5 array with 6 disks (72GB each) - currently nfsen uses 200GB, Red Hat Enterprise Linux 4. Netflow: 2 netflow v9 sources, exporting about 2.5 million flows each every 5 minutes (packet input rate in the collector is about 7Mbps). Currently I have only 3 profiles, and updating them takes about 2 seconds. My plugin that generates traffic statistics based on these exports requires about 200 runs of the nfdump command, and takes about 2.5-3 minutes to run. In the near future (1-2 months) I will have to add at least 4 more sources, 2 of which have an expected data rate of 2Gbps. I know that my collector can handle the load, but I'm not confident that my plugin will be able to finish in time. This is why I had all these questions. For now I will try to add 20 profiles, to see the response time, but, as I said before, I don't believe I will have a big speed increase. Thank you again, and if I misunderstood something, please correct me Adrian Popa PS. I don't want yet a snapshot with shadow channels - I will use it when you are ready to release it for the general public. Peter Haag wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Adrian, > > To add some numbers to my previous mail: > > My nfsen-current developer installation: > > System: HP ProLiant DL360 G3 3GHz > 1 GB RAM > Internal Compaq Smart Array 5i 2x70GB disks mirrored > OS OpenBSD 4.0 > > 2 netflow sources configured, each about 25 - 30MB flows each 5min. > 30 channels to update ( about 12 shadow channels ) > time for profiling: 10s > > Some other plugins - such as PortTracker need much more time. > So there is some room for more channels ... > > - Peter > > - --On February 9, 2007 9:18:50 +0200 Adrian Popa <[EMAIL PROTECTED]> wrote: > > | Peter, I have a question about performance. > | > | If I were to give up on my plugin and instead create about 200 profiles > | , each searching for a different network prefix and input/output > | interface in the flows, would nfsen be able to handle this kind of data? > | (actually I have 20 prefixes on 6 routers with 1 to 4 interfaces each, > | and I'd like to plot upstream and downstream traffic - so I think there > | will be more than 200 profiles...). > | For each profile, I could set an expire time of 10 minutes (because I > | don't need to save the flows - just to get the graphs). > | > | Or, as you suggested, I could use channels in the same profile, but I > | have no idea how I could create or manage them... (perhaps you have some > | tips where I could find some documentation about that). > | > | My main question is: do you think nfsen can handle 200 profiles? I have > | only a production machine, and I'm not eager to experiment on it! :) > | > | Thank you! > | > | Peter Haag wrote: > | > -----BEGIN PGP SIGNED MESSAGE----- > | > Hash: SHA1 > | > > | > Hi Adrian, > | > > | > - --On February 1, 2007 15:28:32 +0200 Adrian Popa <[EMAIL PROTECTED]> > wrote: > | > > | > | Hello, > | > | > | > | I have a question about the performance of nfdump, but first, let me > | > | explain what I'm trying to do: > | > | I have a plugin that searches the collected flows for specific network > | > | prefixes (or AS-es) on each exporting router, on specific intefaces. The > | > | information is then fed into custom rrd files and plotted as png images. > | > | Searching is done by using a top 1 record/bytes and filtering by 'inif x > | > | and net 1.2.3.0/24'. Here's an example: > | > | > | > | $nfdump -r /data/nfsen/profiles/live/$border/nfcapd.$timeslot -n 1 -s > | > | record/bytes -o "fmt:%ts %td %pr %sap -> %dap %pkt %byt %bps %in %out > | > | %sas %das %fl" '$ifType $if and net $prefix' > | > | > | > | I have to search for input traffic on a specific interface for a > | > | specific network prefix and also for output traffic for the same thing. > | > | > | > | I achieved to do this, and it works well, but execution time for 2 > | > | borders (with 3-4 interfaces each), 20 prefixes and 50 AS-es for a peak > | > | traffic of about 2Gbps takes about 3,5 minutes. > | > > | > If I understand you right, you are going to call this nfdump command for > | > each of the prefixes, which results in a lot of sequential nfdump > commands. > | > > | > | > | > | In the future I will want to monitor other routers, on the same > | > | principle. As far as I see, I can do that, but either I gather less > | > | data, or I use a different machine for collecting. > | > | > | > | A colleague of mine proposed that I split my script (which is 100% > | > | sequential) into several threads that run at the same time. Each thread > | > | would call nfdump and update it's particular rrd. > | > | > | > | My question to you is this: Assuming that I start the new processes like > | > | threads (or more likely like forked processes), would I get a speed > | > | increase? I'd like to say that this script keeps the processor usage at > | > | about 60-80%. > | > > | > The CPU usage is only half of the story for your plugin. Almost every time > | > I/O is much more a problem. For each nfdump command you read a lot of > data. > | > If this amount of data does not fit into the file system cache of your OS, > | > the performance rapidly drops. So I'd recommend you to analyse how your > | > system behaves in such a plugin cycle. Have a look at the IO using iostat > | > check the service time of your disks. If you still have room creating > | > more threads can result in better performance. If your IO system is at its > | > limit, you will not gain anything, and CPU will stay at 60-80% as your > system > | > has lots of IO wait. Overcoming this, you would need more RAM to increase > the > | > available memory for the IO file system cache. A high service time in IO > stat > | > means slow disks - so you'll need the right balance of disks and memory. > | > > | > Furthermore you can try to optimise IO by limiting reading data once and > doing > | > parallel processing - the way which nfprofile does profiling all your > channels. > | > It reads the data once only and applies all filters to the same data. > | > > | > Coming back to your plugin - you may try to optimise IO by creating a > profile > | > with a channel per prefix and creating adequate filters per channel. Your > | > plugin then needs to create the stat Top 1 per channel which may result > in reading > | > less data over all - but this is just a guess. > | > > | > So - it's a bit of all. > | > Hope this helps anyway. > | > > | > - Peter > | > > | > | > | > | I don't know if the forked processes would load the same input file into > | > | memory again and again, or if they would share the same file (lowering > | > | memory consumption)? > | > | > | > | What are your recomandations? > | > | > | > | Thank you for your time, > | > | > | > | -- > | > | Adrian Popa > | > | > | > | > | > | > | > | > ------------------------------------------------------------------------- > | > | Using Tomcat but need to do more? Need to support web services, > security? > | > | Get stuff done quickly with pre-integrated technology to make your job > easier. > | > | Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > | > | http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > | > | _______________________________________________ > | > | Nfsen-discuss mailing list > | > | [email protected] > | > | https://lists.sourceforge.net/lists/listinfo/nfsen-discuss > | > > | > > | > > | > - -- > | > _______ SWITCH - The Swiss Education and Research Network ______ > | > Peter Haag, Security Engineer, Member of SWITCH CERT > | > PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7 > | > SWITCH, Limmatquai 138, CH-8001 Zurich, Switzerland > | > E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/ > | > -----BEGIN PGP SIGNATURE----- > | > Version: GnuPG v1.4.3 (Darwin) > | > > | > iQCVAwUBRcMAaP5AbZRALNr/AQLXEwP/bymvl/R3I5MqF8qXSq82QXwDng9VPcyH > | > 56KfUdgDFYpVSOM/Jjn08t8LPaGCA/2DQFxzjzXc+g/YngLfOFZFxkjZEDfRo3AS > | > 53T8cZeTHIx8gy4Xn1y5VqerK0+Q4BNB+I+1+YYo/g8wVfE+pBMNNKh1m+krIwLO > | > isZnG514jMA= > | > =E3HQ > | > -----END PGP SIGNATURE----- > | > > | > > | > > | > | > | ------------------------------------------------------------------------- > | Using Tomcat but need to do more? Need to support web services, security? > | Get stuff done quickly with pre-integrated technology to make your job > easier. > | Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > | http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > | _______________________________________________ > | Nfsen-discuss mailing list > | [email protected] > | https://lists.sourceforge.net/lists/listinfo/nfsen-discuss > > > > - -- > _______ SWITCH - The Swiss Education and Research Network ______ > Peter Haag, Security Engineer, Member of SWITCH CERT > PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7 > SWITCH, Limmatquai 138, CH-8001 Zurich, Switzerland > E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/ > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.3 (Darwin) > > iQCVAwUBRcxHZ/5AbZRALNr/AQKWTQP/XBTsBhVVbh4sJoOnrQlqk1kPlOHx9Elu > J1qEFANATT43zpsa4hwxTitph+shRRFdzOcv5bvuFzrcvJA2extaWKRFll2BCNC5 > RiFNlbER7iuI+PzFt38p0qQf+LSN3Cm0srGaY8IhAXytrEwbgTt+Rjo08Vg6deAN > I7ssmcJzavA= > =G5ny > -----END PGP SIGNATURE----- > > > ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier. Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Nfsen-discuss mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
