I've been doing database-based logfile analysis for about a year and a half at
http://www.techspex.com -- the space requirements are ridiculous, the processing
consumed is monumental, and the information thereby won is absolutely essential to the
management of the company.
If you want really closely customized results, you can't beat database log file
analysis; SQL can just do so many things that are impractical otherwise (duh -- that's
the point of databases, right?)
One thing I do before loading logfiles into the database is preprocess with Perl in
order to remove: graphics hits (who cares?), 404-compliant pages, um, I think that's
it.
The report itself is done with multiple horrible SQL queries, slashed up and glued
together by a very arcane Tcl script (so sue me, when I wrote it I wasn't really into
Perl yet. If I had it to do over I'd likely use Perl. Instead of having it to do
over, however, I have maintenance. :-)
The information we get out of it is extremely customized and there would be no way to
get any existing package to deliver it. (In fact, when Techspex was acquired, one of
the points the president made to me about how cool the move would be is that the new
owners had lots of preexisting log analysis tools so we wouldn't have to worry about
maintaining this stuff. Yeah. WebTrends. I kid you not. This from a guy who's been
poring over his Analog reports *and* detailed custom reports for a year and a half.
Sigh. For a non-pointy-haired guy he sure can believe in easy solutions.)
That's my two bits. You're welcome to ask more questions.
Jeremy Wadsack wrote:
> Alejandro Fernandez wrote:
>
> > Also, here at Sift we're starting to parse logfiles into a database, so we can
> > work "group by" magic on them to do with all the other data we have. Anyone
> > else done that?
>
> I'd be interested to know your experience on this. Mine has been that logfiles
> bring database servers to their knees. You'll need a fairly powerful database
> server to handle this (Oracle, Sybase,...). And something to consider is that with
> logfiles, you don't really care about the records, you just care about the overall
> picture they present. Also, the space requirement will be HUGE: databases usually
> incure 2 - 5 times the (uncompressed) flatfile storage space.
>
> On the otherhand, a database gives you a lot more flexibility in generating reports
> (SQL on logfiles!).
>
> HTH,
>
> Jeremy Wadsack
> Wadsack-Allen Digital Group
>
> ------------------------------------------------------------------------
> This is the analog-help mailing list. To unsubscribe from this
> mailing list, send mail to [EMAIL PROTECTED]
> with "unsubscribe" in the main BODY OF THE MESSAGE.
> List archived at http://www.mail-archive.com/[email protected]/
> ------------------------------------------------------------------------
------------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe" in the main BODY OF THE MESSAGE.
List archived at http://www.mail-archive.com/[email protected]/
------------------------------------------------------------------------
