No. This definitely won\'t work. I am not aware of any
database that can (a) handle storage better than
gzipping log files or (b) crunch them faster than
Analog (but feel free to correct me if I\'m wrong).
Analog is designed to do logfile crunching and only
that. It works very efficiently. SQL databases on the
other hand are designed to do a lot of things and are
not as efficient.
Unfortunately, a lot of people seem to think this would
be more efficient. I have brought enterprise database
servers to their knees with smaller logfiles than the
ones this thread was about.
Anyway, this being analog-help, I think I\'ll stick with
analog. :) But thanks for the suggestion.
Jeremy Wadsack
Wadsack-Allen Digital Group
Quoting Vibol Hou <[EMAIL PROTECTED]>:
> Hi Jeremy,
>
> Since going through enormous amounts of flat files is
at best haphazard,
> your best bet is to place all the log information into
a SQL database.
> There is a Perl script being developed which will
allow you to insert
> logfile entries into a SQL database. You can find it
here:
> http://www.goofy.gaudi.dhs.org/en/apachedb.php3.
>
> This means you won\'t be able to use analog to process
the log entries
> anymore unless analog is modified to support DBs.
>From there, you can
> write
> the appropriate scripts that perform whatever queries
are required from the
> database.
>
> -Vibol
>
> -----Original Message-----
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED]]On Behalf Of
> [EMAIL PROTECTED]
> Sent: Thursday, March 02, 2000 3:48 PM
> To: [EMAIL PROTECTED]
> Subject: RE: [analog-help] Huge Logfiles
>
>
> I don\\\'t know how big they are. This configuration has
> about a dozen web servers each of which creates a new
> logfile every hour. They are all stored in a single
> directory stating over a year ago. So the server
won\\\'t
> even show the directory listing. (Suppose I could
> do \\\'du\\\' a level below or somthing...) But perhaps as
a
> benchmark, dnstran took three days to build a cache
> file of the first year\\\'s logs....
>
> Unfortunately, marketing want stats, systems want
> stats, development wants stats ... add it all up and
we
> can\\\'t really cut anything out. We may, though, split
> the runs and create several sets of smaller cache
files
> for each target so they can be handled.
>
> In the meantime we\\\'ve moved to a 14GB system for the
> time being. But any more thoughts on improving
> efficiency would be appreciated.
>
> Thanks,
>
> Jeremy Wadsack
> Wadsack-Allen Digital Group
>
>
> Quoting Shakeel Sorathia <[EMAIL PROTECTED]>:
>
> > Holy Pete, how big are your daily log files! We
used
> to run into the same
> > problem a while back with some of our sites. It
> would eat up all the ram
> > on
> > the box to do a report. What I did was the
> following. I wrote a one line
> > perl script to pre-process the log files to strip
> them of all .gifs and
> > .jpgs. We don\\\'t use that info anyways so I decided
> to take them out of the
> > log files. This helped to shrink down the size of
> the log files, as well
> > as
> > the amount of ram that analog required to run stats
> on them.
> >
> > --shak
> >
> > Shakeel Sorathia
> > Unix Team
> > [EMAIL PROTECTED]
> > 626-660-3502
> >
> > > -----Original Message-----
> > > From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED]]
> > > Sent: Thursday, March 02, 2000 2:29 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: [analog-help] Huge Logfiles
> > >
> > >
> > > In order to process some large logfiles, we\\\\\\\'re
> running
> > > daily jobs with caches and then reprocessing the
> cache
> > > to weekly and monthly stats. However, the daily
> cache
> > > files are running about 100 - 300 MB each. Running
a
> > > weekly report uses about 1.5GB of system memory on
> the
> > > server (about all that machine has). And we
> haven\\\\\\\'t got
> > > to monthly yet...
> > >
> > > Two questions for anyone with advice:
> > >
> > > 1] Is it safe to assume that the memory usage for
> > > analog to process a set of cache files will be
about
> > > the total of the size of those files? (e.g. the
> weekly
> > > report used 1.5GB -- the sum of all the cache file
> > > sizes).
> > >
> > > 2] Does anyone have advice as to how to reduce the
> > > memory usage (yes, we already have HOSTLOWMEM 2
and
> > > FILELOWMEM 2)? Do we need to turn off some
reports?
> > > Does *LOWMEM 3 make a difference when processing
> from a
> > > cachefile? Can we reduce the data in the files?
Will
> > > gzipping the files reduce the overall memory usage
> by
> > > analog?
> > >
> > > Should we just run it on a 28GB box?
> > >
> > > Thanks,
> > >
> > > Jeremy Wadsack
> > > Wadsack-Allen Digital Group
> > >
----------------------------------------------------
> ----------
> > > ----------
> > > This is the analog-help mailing list. To
> unsubscribe from this
> > > mailing list, send mail to analog-help-
> [EMAIL PROTECTED]
> > > with \\\"unsubscribe\\\" in the main BODY OF THE
MESSAGE.
> > > List archived at
> > http://www.mail-archive.com/analog-
> [EMAIL PROTECTED]/
> >
------------------------------------------------------
> ------------------
> >
------------------------------------------------------
> ------------------
> > This is the analog-help mailing list. To unsubscribe
> from this
> > mailing list, send mail to analog-help-
> [EMAIL PROTECTED]
> > with \\\"unsubscribe\\\" in the main BODY OF THE
MESSAGE.
> > List archived at http://www.mail-archive.com/analog-
> [EMAIL PROTECTED]/
> >
------------------------------------------------------
> ------------------
> >
>
>
--------------------------------------------------------
----------------
> This is the analog-help mailing list. To unsubscribe
from this
> mailing list, send mail to
[EMAIL PROTECTED]
> with \"unsubscribe\" in the main BODY OF THE MESSAGE.
> List archived at
http://www.mail-archive.com/[email protected]/
>
--------------------------------------------------------
----------------
>
>
--------------------------------------------------------
----------------
> This is the analog-help mailing list. To unsubscribe
from this
> mailing list, send mail to
[EMAIL PROTECTED]
> with \"unsubscribe\" in the main BODY OF THE MESSAGE.
> List archived at
http://www.mail-archive.com/[email protected]/
>
--------------------------------------------------------
----------------
>
------------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe" in the main BODY OF THE MESSAGE.
List archived at http://www.mail-archive.com/[email protected]/
------------------------------------------------------------------------
