Hi Dave,
Thanks again for responding. I had a look at the rc scripts, I did notice flow-capture had 2 startup scripts, which was wierd. I removed that. But not flowscan, it's only being started once. Here's a top with the required info. top - 16:04:06 up 1 day, 5:34, 1 user, load average: 9.20, 8.68, 8.46 Tasks: 83 total, 7 running, 75 sleeping, 0 stopped, 1 zombie Cpu0 : 96.0%us, 4.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 98.0%us, 2.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1036428k total, 1022864k used, 13564k free, 17768k buffers Swap: 5301368k total, 52k used, 5301316k free, 758620k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ PPID P SWAP TIME COMMAND 9013 root 18 0 30280 26m 1732 R 44 2.7 2:38.67 1 0 2776 2:38 flowscan 16158 root 18 0 30284 26m 1732 R 44 2.7 0:51.91 1 1 2780 0:51 flowscan 24721 root 18 0 30116 26m 1728 R 40 2.6 6:12.83 1 0 2656 6:12 flowscan 3869 root 18 0 30120 26m 1728 R 39 2.6 3:51.55 1 0 2656 3:51 flowscan 2384 mysql 15 0 128m 36m 5220 S 13 3.6 252:44.53 2347 0 91m 252:44 mysqld 29815 root 23 0 30284 26m 1732 R 9 2.7 5:44.42 1 1 2780 5:44 flowscan Regards, Ross From: Dave Plonka [mailto:[EMAIL PROTECTED] Sent: Thursday, 21 February 2008 3:38 PM To: [email protected] Cc: [EMAIL PROTECTED]; Ross Tsolakidis; [EMAIL PROTECTED] Subject: Re: [Flow-tools] cpu utilisation on flowscan processes Hi Ross, I'm Cc'ing this to [EMAIL PROTECTED] since that is the flowscan-specific mailing list. I've replied in context below: On Thu, Feb 21, 2008 at 02:13:15PM +1100, Ross Tsolakidis wrote: > Hi all, > > I just needed some clarification on whether this is normal. > We run flowscan with flow-tools then extract the data from rrd into a mysql DB for usage figures. > > One thing I'm noticing though is it's very CPU intensive. <snip> > Flowscan runs every 5 minutes, using the CUFlow class. > > 2008/02/21 07:20:07 working on file /var/netflow/ft-v05.2008-02-21.071500+1100... > 2008/02/21 07:20:10 flowscan-1.020 CUFlow: Cflow::find took 3 wallclock secs ( 3.36 usr + 0.00 sys = 3.36 CPU) for 585896 flow file bytes, flow hit ratio: 32073/32970 > 2008/02/21 07:20:11 flowscan-1.020 CUFlow: report took 1 wallclock secs ( 0.00 usr 0.00 sys + 0.11 cusr 0.04 csys = 0.15 CPU) OK, looks good (more details below). > The CUFlow.cf has approx 31 Class Cs in it. > I am analysing every IP, eg, every IP in those Class Cs has it's own RRD. > Not sure if this is too much for it to do. Not at all. > 2336 ? Ss 0:37 /usr/bin/flow-capture -w /var/netflow/ft 0/0/2055 -S5 -V5 -E1G -n 287 -N 0 -R /usr/local/netflow/bin/linkme > 3099 ? S 4:20 /usr/bin/perl /usr/bin/flowscan > 4941 ? R 1:31 /usr/bin/perl /usr/bin/flowscan > > As you can see it???s running 2 sometimes 3 processes of flowscan. > Is this normal ? No, it's not normal to have two flowscan processes... probably a mistake with your rc scripts, i.e. started it twice. I'd kill off the older one. (Of course look at the PPID first, to verify they are unrelated.) > Am I doing this right ? ??? > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 4941 root 25 0 21708 18m 1700 R 100 0.9 3:19.58 flowscan > 2336 root 15 0 2896 1244 504 S 1 0.1 0:37.41 flow-capture > > I missed the 2nd flowscan in top while writing this email, > but it basically flatlines the cpu 24/7 doing this. Doing what? Your flowscan log shows that it only worked for 4 seconds to process 5 minutes of flows. If you haven't cleared it up after this email, I suggest telling us the load average on the machine. If there's only <5 seconds of CPU-intensive flowscan work every five minutes, it should be golden. If the load average is high, then perhaps there's something unrelated wrong with your machine, in which case sar will get you far... (Look at running sadc to collect performance information.) > Can someone point out anything that I am doing wrong ? >From what you've shown in the log and ps output, it looks good to me ('cept the two flowscan processes, but that should at most incorrectly double the load). To reiterate, it's only taking a few seconds for the find and report phases (shown as 3 and 1 wallclock seconds, respecively, in the flowscan logs above) for you to process 5 minutes of flow data. So, as soon as everything is caught up (so that it's processing in real time, flowscan will only be hitting the CPU for about 5 seconds every five minutes... As for what is using the CPU inside flowscan, it is the Cflow perl module (that translates the flow files for your perl script) and the report code (in your case CUFlow) that uses the CPU - its because it's perl code... lots of management of data structures and also a lot of converting raw flow records (from the flow files) into perl variables. Dave -- [EMAIL PROTECTED] http://net.doit.wisc.edu/~plonka/ Madison, WI ________________________________ Total Control Panel Login <https://asp-6.reflexion.net/login?domain=day3.com.au> To: [EMAIL PROTECTED] <https://asp-6.reflexion.net/address-properties?aID=6477361> From: [EMAIL PROTECTED] <https://asp-6.reflexion.net/address-properties?aID=31239355> Message Score: 50 High (60): Pass My Spam Blocking Level: Medium Medium (75): Pass Low (90): Pass Block <https://asp-6.reflexion.net/FooterAction?ver=1&bl-sender-address=1&rID= 6477361&aID=31239355> this sender / Block <https://asp-6.reflexion.net/FooterAction?ver=1&ent=1&bl-sender-address= 1&rID=6477361&aID=31239355> this sender enterprise-wide Block <https://asp-6.reflexion.net/FooterAction?ver=1&bl-sender-domain=1&rID=6 477361&aID=31239355> doit.wisc.edu / Block <https://asp-6.reflexion.net/FooterAction?ver=1&ent=1&bl-sender-domain=1 &rID=6477361&aID=31239355> doit.wisc.edu enterprise-wide This message was delivered because the content filter score did not exceed your filter level. DISCLAIMER: This e-mail and any files transmitted with it may be privileged and confidential, and are intended only for the use of the intended recipient. If you are not the intended recipient or responsible for delivering this e-mail to the intended recipient, any use, dissemination, forwarding, printing or copying of this e-mail and any attachments is strictly prohibited. If you have received this e-mail in error, please REPLY TO the SENDER to advise the error AND then DELETE the e-mail from your system. Any views expressed in this e-mail and any files transmitted with it are those of the individual sender, except where the sender specifically states them to be the views of our organisation. Our organisation does not represent or warrant that the attached files are free from computer viruses or other defects. 21/2/2008 The user assumes all responsibility for any loss or damage resulting directly or indirectly from the use of the attached files. In any event, the liability to our organisation is limited to either the resupply of the attached files or the cost of having the attached files resupplied.
_______________________________________________ Flow-tools mailing list [EMAIL PROTECTED] http://mailman.splintered.net/mailman/listinfo/flow-tools
