You can also do things like compress the log files to save space. Analog (properly configured) can decompress the logs on the fly for analysis. (Very cool) We are analyzing about 300MB a day. After gzip-ing the logs as they are downloaded, I reduce my storage to about 18MB per day. The machine I built for the logs was a P-III/850 with 768MB of RAM and after only 8 weeks of data, I was running into memory problems. I used the LOMEM tricks that Jeremy mentioned below, but I don't think that you are going to be able to get 6 months of data to crunch, unless you have something you can put a lot of RAM into.
----- Original Message ----- From: <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Monday, January 07, 2002 12:45 PM Subject: Re: [analog-help] Large logfiles Roy Iversen ([EMAIL PROTECTED]): > Situation: We're currently hosting a news-service for a client on > IIS5.0, and they now require better statistics. > -The site produces one log file each day, which is about 400-500 megs in > size. > -The client would like to see statistics for the past 6 months. > This would make up something like 82 GB of logs which needs to be > analyzed each week, preferably each night, to produce the report. Of > course this is almost an impossible task. > I'm new to Analog, so I hope there is something I've missed, because I > really like Analog + ReportMagic. But unfortunately it seems to fit only > for small sites. > Any ideas ? Start with cache files: http://www.analog.cx/docs/cache.html. After you get that working, if you still have issue with memory, look at the *LOWMEM commands: http://www.analog.cx/docs/lowmem.html. If that doesn't help, try adding more memory (it's cheap) or consider breaking up reports (e.g. only offer monthly reports, or only look at subsections of the data). -- Jeremy Wadsack Wadsack-Allen Digital Group +------------------------------------------------------------------------ | This is the analog-help mailing list. To unsubscribe from this | mailing list, go to | http://lists.isite.net/listgate/analog-help/unsubscribe.html | | List archives are available at | http://www.mail-archive.com/[email protected]/ | http://lists.isite.net/listgate/analog-help/archives/ | http://www.tallylist.com/archives/index.cfm/mlist.7 +------------------------------------------------------------------------ +------------------------------------------------------------------------ | This is the analog-help mailing list. To unsubscribe from this | mailing list, go to | http://lists.isite.net/listgate/analog-help/unsubscribe.html | | List archives are available at | http://www.mail-archive.com/[email protected]/ | http://lists.isite.net/listgate/analog-help/archives/ | http://www.tallylist.com/archives/index.cfm/mlist.7 +------------------------------------------------------------------------
