On Sun, 26 Jan 2003, Graham Mercer wrote: > Sorry for what must be a dumb question but I am VERY new to web logs! I have > just created a website (www.potteryexpo.com) from which I d/l raw log data. > The site has only been up for a week and already the log is 1.4MB. > Is there a way that I can periodically save this log data in reasonably > sized chunks and still be able to get a comprehensive analysis from the data > over a long period of time? > Seems to me that at the end of a month the log is going to be around 10 MB, > which is way too much for my 56K dial-up to be regularly d/ling. Similarly > to keep the log data for a year it would be over 120MB??? >
Disk space is pretty cheap now, so that shouldn't be too much of a concern. But bandwidth to download it over a regular modem could be a problem. If you want to process the logfiles on your own machine, at some point you're going to have to download them. You only need to download the portion which is new, though, not the whole of the historical data every time. (The alternative is to do some processing on the server first, which is possible, but would require the cooperation of the server administrators.) One important point: is the log data compressed? I.e., are the files plain text, or are they zipped or gzipped? If they're not compressed, you can usually get them down to about 1/10th the size with compression, and your server administrators should be happy to compress them for you. This would probably solve your problems, at least for the moment. -- Stephen Turner, Cambridge, UK http://homepage.ntlworld.com/adelie/stephen/ "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." (Edsger W. Dijkstra) +------------------------------------------------------------------------ | TO UNSUBSCRIBE from this list: | http://lists.isite.net/listgate/analog-help/unsubscribe.html | | Digest version: http://lists.isite.net/listgate/analog-help-digest/ | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general | List archives: http://www.analog.cx/docs/mailing.html#listarchives +------------------------------------------------------------------------
