OK, after more hours of testing....
I undid a bunch of my tweaks to globals-defines.h and recompiled everything, same problem. RAM usage was far less as expected, but cpu still maxed and netflow queues always full. I recompiled my 3.2.1 stuff with the same tweaks and it runs perfect, queues empty almost immediately after startup and my cpu is averaging about 15% right now, load avg is .58! I did notice this instance is using 20 threads on 3.2.1 whereas it was only.... 13 or so on 3.3.9. On 3.3.9 I've: undef'd HAVE_SNMP, disabled rrd, and tried to disable GeoIP by renaming the .dat files. Any help would be appreciated - spent at least 40 hours on this so far! Thanks! Gary ________________________________ From: Gary Gatten Sent: Tuesday, May 12, 2009 3:57 PM To: '[email protected]'; '[email protected]' Subject: RE: high CPU load / can't keep up with netflow since upgrade to 3.3.8 PS: On 3.3.9 from SourceForge now - got tired of the "[warn] kevent: Bad file descriptor" errors. G ________________________________ From: Gary Gatten Sent: Tuesday, May 12, 2009 3:56 PM To: '[email protected]'; '[email protected]' Subject: RE: high CPU load / can't keep up with netflow since upgrade to 3.3.8 OK, under production load I'm having similar issues even with "HAVE_SNMP" undef'd. I tweaked some timers in globals-defines.h that is causing higher than typical memory usage. I don't have much RAM (768MB) so I'm doing a lot of swapping. This is part of the problem for sure, so at this point I'm not 100% sure if the SNMP (UtilLoop) had anything to do with this and if so how much? I do recall when first trying to get on 3.3.x some time ago I didn't tweak the timers as much, but I also had even less RAM then! More testing, but prolly not today! ________________________________ From: Gary Gatten Sent: Monday, May 11, 2009 6:42 PM To: '[email protected]'; '[email protected]' Subject: RE: high CPU load / can't keep up with netflow since upgrade to 3.3.8 Thanks for all the prompt replies on this! :-) Looks like the additional load is related to SNMP. I commented out and undef'd stuff related to "HAVE_SNMP" and SO FAR my CPU load is back to "normal" and the udp queues are being serviced promptly. Not a whole lot of traffic right now though, so will let you know for sure EOD tomorrow - I know you're all anxiously waiting! G ________________________________ From: Gary Gatten Sent: Monday, May 11, 2009 3:27 PM To: '[email protected]'; '[email protected]' Subject: RE: high CPU load / can't keep up with netflow since upgrade to 3.3.8 What's diff between 3.2.x and 3.3.x netflow plugin? I deleted the prefsCache.db and started adding netflow devices 1 at a time. Each device as between 10 and 30 netflow exporters, and not high volume ones; all T1 sites with ~ 15 users each. Anyway, all is going well until I hit 3 or 4 netflow devices, then cpu maxes out and queues start filling up. Number of flows / sec / min is as stable as a production environment can be - not drastic changes that would account for this high of load. If there something different with the threads? It seems to be acting differently - just a hunch. ________________________________ From: Gary Gatten Sent: Monday, May 11, 2009 11:08 AM To: '[email protected]'; [email protected] Subject: high CPU load / can't keep up with netflow since upgrade to 3.3.8 Been running 3.2.1 for years and CPU has always been fine. Avg, load on FreeBSD 6.0 is around .5, CPU around 40%, and udp port queues (netstat -a) are most always zero, but sometimes a couple get a little backed up for a few mins here and there. Now running 3.3.8 compiled from ports. Load now 1.8, CPU 100%, and most all udp netflow queues are maxed and not emptying, especially within one instance that has many flow exporters - about 160'ish: $ netstat -a | grep flow Ntop Instance 1 udp4 41200 0 *.netflow-Regn29 *.* udp4 41496 0 *.netflow-Regn16 *.* udp4 41456 0 *.netflow-Regn15 *.* udp4 40904 0 *.netflow-Regn11 *.* udp4 41544 0 *.netflow-Regn10 *.* udp4 41456 0 *.netflow-Regn02 *.* udp4 41552 0 *.netflow-Regn88 *.* udp4 40656 0 *.netflow-Regn53 *.* Ntop Instance 2 udp4 0 0 *.netflow-Inet *.* Ntop Instance 3 udp4 0 0 *.netflow-LANCore *.* 3.2.1 had its issues, but as far as netflow and cpu load it was fine. I'd like to get on a more recent version, but keep hitting roadblocks. Tried this back with 3.3.1 (I think?) with similar results. Nothing in logs I find interesting. Only interesting thing on compile what something about "sched.h": checking sys/sched.h presence... yes configure: WARNING: sys/sched.h: present but cannot be compiled configure: WARNING: sys/sched.h: check for missing prerequisite headers? configure: WARNING: sys/sched.h: see the Autoconf documentation configure: WARNING: sys/sched.h: section "Present But Cannot Be Compiled" configure: WARNING: sys/sched.h: proceeding with the preprocessor's result configure: WARNING: sys/sched.h: in the future, the compiler will take precedence I'm willing to do anything reasonable to help get this resolved. Any help would be greatly appreciated!!! TIA! Gary <font size="1"> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in 1.0pt 0in'> </div> "This email is intended to be reviewed by only the intended recipient and may contain information that is privileged and/or confidential. If you are not the intended recipient, you are hereby notified that any review, use, dissemination, disclosure or copying of this email and its attachments, if any, is strictly prohibited. If you have received this email in error, please immediately notify the sender by return email and delete this email from your system." </font>
_______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
