I don\'t know how big they are. This configuration has
about a dozen web servers each of which creates a new
logfile every hour. They are all stored in a single
directory stating over a year ago. So the server won\'t
even show the directory listing. (Suppose I could
do \'du\' a level below or somthing...) But perhaps as a
benchmark, dnstran took three days to build a cache
file of the first year\'s logs....
Unfortunately, marketing want stats, systems want
stats, development wants stats ... add it all up and we
can\'t really cut anything out. We may, though, split
the runs and create several sets of smaller cache files
for each target so they can be handled.
In the meantime we\'ve moved to a 14GB system for the
time being. But any more thoughts on improving
efficiency would be appreciated.
Thanks,
Jeremy Wadsack
Wadsack-Allen Digital Group
Quoting Shakeel Sorathia <[EMAIL PROTECTED]>:
> Holy Pete, how big are your daily log files! We used
to run into the same
> problem a while back with some of our sites. It
would eat up all the ram
> on
> the box to do a report. What I did was the
following. I wrote a one line
> perl script to pre-process the log files to strip
them of all .gifs and
> .jpgs. We don\'t use that info anyways so I decided
to take them out of the
> log files. This helped to shrink down the size of
the log files, as well
> as
> the amount of ram that analog required to run stats
on them.
>
> --shak
>
> Shakeel Sorathia
> Unix Team
> [EMAIL PROTECTED]
> 626-660-3502
>
> > -----Original Message-----
> > From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]]
> > Sent: Thursday, March 02, 2000 2:29 PM
> > To: [EMAIL PROTECTED]
> > Subject: [analog-help] Huge Logfiles
> >
> >
> > In order to process some large logfiles, we\\\'re
running
> > daily jobs with caches and then reprocessing the
cache
> > to weekly and monthly stats. However, the daily
cache
> > files are running about 100 - 300 MB each. Running a
> > weekly report uses about 1.5GB of system memory on
the
> > server (about all that machine has). And we
haven\\\'t got
> > to monthly yet...
> >
> > Two questions for anyone with advice:
> >
> > 1] Is it safe to assume that the memory usage for
> > analog to process a set of cache files will be about
> > the total of the size of those files? (e.g. the
weekly
> > report used 1.5GB -- the sum of all the cache file
> > sizes).
> >
> > 2] Does anyone have advice as to how to reduce the
> > memory usage (yes, we already have HOSTLOWMEM 2 and
> > FILELOWMEM 2)? Do we need to turn off some reports?
> > Does *LOWMEM 3 make a difference when processing
from a
> > cachefile? Can we reduce the data in the files? Will
> > gzipping the files reduce the overall memory usage
by
> > analog?
> >
> > Should we just run it on a 28GB box?
> >
> > Thanks,
> >
> > Jeremy Wadsack
> > Wadsack-Allen Digital Group
> > ----------------------------------------------------
----------
> > ----------
> > This is the analog-help mailing list. To
unsubscribe from this
> > mailing list, send mail to analog-help-
[EMAIL PROTECTED]
> > with \"unsubscribe\" in the main BODY OF THE MESSAGE.
> > List archived at
> http://www.mail-archive.com/analog-
[EMAIL PROTECTED]/
> ------------------------------------------------------
------------------
> ------------------------------------------------------
------------------
> This is the analog-help mailing list. To unsubscribe
from this
> mailing list, send mail to analog-help-
[EMAIL PROTECTED]
> with \"unsubscribe\" in the main BODY OF THE MESSAGE.
> List archived at http://www.mail-archive.com/analog-
[EMAIL PROTECTED]/
> ------------------------------------------------------
------------------
>
------------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe" in the main BODY OF THE MESSAGE.
List archived at http://www.mail-archive.com/[email protected]/
------------------------------------------------------------------------
