Scheidel, Greg (Contractor) wrote:
I'm running a pretty general 'log processing' Perl script, where I open log files one at a time, read the lines, parse them and update counters, then close and read the next log file. The hash I'm storing data in can grow over time, as new users are found and added to the tally list. The number of users will tend to grow quickly at first, then level out as the same users are seen over and over again.
The problem is that I seem to be hitting a limit where the script (nearly) grinds to a halt. It's still going... but very slowly. I'm almost certain that I'm looking at an OS or ActivePerl issue, but I'm querying here to try and make sure I'm not missing anything.
Consistently when I get to ~2GB worth of data read, the loop 'ticker' slows down to intermittent activity... nothing for minutes at a time, a burst of a few hundred lines read and processed, then nothing for a few more minutes.
This is a dual CPU Windows box. CPU utilization on processors 1 and 2 are almost *exact* mirror images of one another, with the signs reversed centering around 50%. In other words, if one is at 75% the other is at 25%. If one is at 80% the other is at 20%. Etc. This keeps average CPU utilization almost exactly on 50% (average 51.499%, max 100%). Over a period of 10 seconds a single CPU might jump from 85% to 25% to 70% to 40% to 60%... up / down pretty consistently.
At the same time, context switches/sec hold pretty consistently around 4000. But the max is ~7000.
Now all the while that this is going on I can see that the memory usage for the perl process is consistently growing. Sometimes 4 Kbytes, sometimes nothing for a few seconds, sometimes 100Kbytes per sec.
I can also see (by the ticker that I have in the script) that there activity in the script (i.e. the loop to read and process log lines) is very sporadic... it sits for several minutes at a time doing nothing, then processes a couple hundred lines, then does nothing for a few more minutes again.
The Perl process itself... interesting. % processor time is pretty well pegged at 99 to 100%. Which makes sense; it's a dual CPU box, the overall CPU usage is pretty firmly averaging 50%, therefore the process' % procesor time is going to be 100%. Handle count is solidly at 22. IO writes is 0. IO reads hover between 2 and 5, but average at 4. Page file bytes is pinned at 173,815,876... with an occassional jump in the max, where it then stays pinned at. Virtual bytes is pinned at 311,595,008, with an occassional jump in the max, where it then stays pinned at. Virtual bytes max (perhaps obviously) is the same as Virtual bytes, likeway page file bytes max. Pool page bytes and Non-pool page bytes are almost identical at ~ 1,332,000.
Now then... every once in a while (sometimes once every 2 minutes, sometimes twice in 30 seconds) there is a peak where context switching and CPU utilization on both switches maxes out. I cannot draw a direct correlation between that peak and subsequent process memory utilization or script activity; in fact, I've observed instances where the script kicks in just after, and instances where it just sits there. Doesn't seem to be a direct relationship.
Is this an obvious paging issue, or problem with ActiveState Perl? Or is this an obscure (or not so obscure, I don't claim to be a Perl expert) Perl issue? Either way, any suggestions on what to do about it (besides switching to a database backend)?
Any and all suggestions and comments are greatly appreciated.
Greg S. _______________________________________________ ActivePerl mailing list [EMAIL PROTECTED] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
_______________________________________________ ActivePerl mailing list [EMAIL PROTECTED] To unsubscribe: http://listserv.ActiveState.com/mailman/mysubs
