Holy Pete, how big are your daily log files! We used to run into the same
problem a while back with some of our sites. It would eat up all the ram on
the box to do a report. What I did was the following. I wrote a one line
perl script to pre-process the log files to strip them of all .gifs and
.jpgs. We don't use that info anyways so I decided to take them out of the
log files. This helped to shrink down the size of the log files, as well as
the amount of ram that analog required to run stats on them.
--shak
Shakeel Sorathia
Unix Team
[EMAIL PROTECTED]
626-660-3502
> -----Original Message-----
> From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, March 02, 2000 2:29 PM
> To: [EMAIL PROTECTED]
> Subject: [analog-help] Huge Logfiles
>
>
> In order to process some large logfiles, we\'re running
> daily jobs with caches and then reprocessing the cache
> to weekly and monthly stats. However, the daily cache
> files are running about 100 - 300 MB each. Running a
> weekly report uses about 1.5GB of system memory on the
> server (about all that machine has). And we haven\'t got
> to monthly yet...
>
> Two questions for anyone with advice:
>
> 1] Is it safe to assume that the memory usage for
> analog to process a set of cache files will be about
> the total of the size of those files? (e.g. the weekly
> report used 1.5GB -- the sum of all the cache file
> sizes).
>
> 2] Does anyone have advice as to how to reduce the
> memory usage (yes, we already have HOSTLOWMEM 2 and
> FILELOWMEM 2)? Do we need to turn off some reports?
> Does *LOWMEM 3 make a difference when processing from a
> cachefile? Can we reduce the data in the files? Will
> gzipping the files reduce the overall memory usage by
> analog?
>
> Should we just run it on a 28GB box?
>
> Thanks,
>
> Jeremy Wadsack
> Wadsack-Allen Digital Group
> --------------------------------------------------------------
> ----------
> This is the analog-help mailing list. To unsubscribe from this
> mailing list, send mail to [EMAIL PROTECTED]
> with "unsubscribe" in the main BODY OF THE MESSAGE.
> List archived at
http://www.mail-archive.com/[email protected]/
------------------------------------------------------------------------
------------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe" in the main BODY OF THE MESSAGE.
List archived at http://www.mail-archive.com/[email protected]/
------------------------------------------------------------------------
