Jin, thanks for your suggestions. I just received another 'job done' notice, from
running analog with *LOWMEM 3. It did reduce times significantly:
Command being timed: "/opt/analog/analog-5.32/analog
+g/opt/analog/conf.d/wwwinfo.analog.cfg"
User time (seconds): 3546.54
System time (seconds): 119.56
Percent of CPU this job got: 102%
Elapsed (wall clock) time (h:mm:ss or m:ss): 59:19.70
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 0
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 150566
Minor (reclaiming a frame) page faults: 42510
Voluntary context switches: 0
Involuntary context switches: 0
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
>From 5 hours to under 1.
The report even looks useful, too.
I'm trying to run out the door, so I'll take some time Monday to analyze things and
consider your suggestions and how to implement them. Thanks, again.
You all have a good weekend.
-Kevin
>>> [EMAIL PROTECTED] 10/15/04 04:56PM >>>
KEVIN ZEMBOWER wrote:
>In my next trial, I'm going to set all the *LOWMEM commands to 3, just to see the
>effect. Any other suggestions to try? No portion of the latest report seems overly
>long.
>
>
I guess that will give you longer execution time. :)
>Stephen originally suggest that my long run times were due to the system thrashing;
>that's how I started with the LOWMEM commands. Is there any other way to increase the
>memory allocation for this program in Unix? I don't think 'nice' has anything to do
>with this. Any compiler options? Anything I'm not thinking of?
>
>
For most 32 bit systems, 2GB is the limit. If your total physical RAM is
much more than 2GB, thrashing should not be a problem. (My system has
4GB RAM and I don't worry about system thrashing). I am not aware of any
increasing memory allocation techniques for a single process program on
a 32 bit system. That means analog can maximally use 2GB RAM on most
systems (latest Linux kernel may be able to assign 3GB process memory on
32 bit system).
Here is a joke; If Stephen bring out a new version of analog which can
do parallel log analyzing based on a forked multi-process model, we may
have analog use as much as memory our machine can support. :)
Turning on LOWMEM can prevent analog itself from running out of the 2GB
virtual memory limit, but will increase the running time without a
doubt. Before you figureout which parts of the reporting is eating CPU
time, I suggest completely turning off some report features, say
REQUEST, DOMAIN, HOST and/or DIRECTORY. After you get running time
significantly dropped, you can turn them back on one by one and tweak
those FLOOR values to get your acceptable result. This way you can
quickly figure out which part is the beast eating running time and tweak
it properly.
As to those LOWMEM parameters, as long as analog does not crash and
system is not swapping. You don't need set all of them to 3.
Jin
>Thanks for your suggestions.
>
>-Kevin
>
>
> Command being timed: "/opt/analog/analog-5.32/analog
> +g/opt/analog/conf.d/wwwinfo.analog.cfg"
> User time (seconds): 1426.31
> System time (seconds): 740.86
> Percent of CPU this job got: 10%
> Elapsed (wall clock) time (h:mm:ss or m:ss): 5:31:57
> Average shared text size (kbytes): 0
> Average unshared data size (kbytes): 0
> Average stack size (kbytes): 0
> Average total size (kbytes): 0
> Maximum resident set size (kbytes): 0
> Average resident set size (kbytes): 0
> Major (requiring I/O) page faults: 3222935
> Minor (reclaiming a frame) page faults: 1571733
> Voluntary context switches: 0
> Involuntary context switches: 0
> Swaps: 0
> File system inputs: 0
> File system outputs: 0
> Socket messages sent: 0
> Socket messages received: 0
> Signals delivered: 0
> Page size (bytes): 4096
> Exit status: 0
>
>
>+------------------------------------------------------------------------
>| TO UNSUBSCRIBE from this list:
>| http://lists.meer.net/mailman/listinfo/analog-help
>|
>| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
>| List archives: http://www.analog.cx/docs/mailing.html#listarchives
>+------------------------------------------------------------------------
>
>
>
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
+------------------------------------------------------------------------
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
+------------------------------------------------------------------------
