Hi,
I'm currently writing a big .sh file to take the stress off of me and to please
the execs with pretty graphs of out 20 or so sites. The idea is to use pipes
the way they were meant, to parse the logs as quickly as possible, and all in
commands like the following:
zcat `find /home1/logs/sift/*poopoo* -name '*transfer-log*gz' -mtime -45` |
analog +ganalog/poopoo.cfg +C"FROM 990701:0000" +C"TO 990801:0000"
+C"FILEINCLUDE /eandt/*" rmagic.pl poopoo.ini
mv /software/apache/htdocs/poopoo/alpha/stats/output/* /software/apache/htdocs
/poopoo/alpha/stats/eandt/july/
It may look like a big lump of unintelligible pipes, but to me it's a work of
art.
This zcats only gzipped log files going back 45 days ago, to produce report.dat
of the month of august. All the lines are the same as this, in that the report
magic stuff goes to the "output" directory and then gets copied wherever it
needs to go. It should also only give results for the eandt directory. But it's
not understanding that "FILEINCLUDE /eandt/*" command, and it's giving me all
the directories in the site.
Oh help me please! :(
Thank you,
Ale
On Thu, 05 Aug 1999, you wrote:
>On Wed, 4 Aug 1999, Sue wrote:
>
>>
>> Hello,
>>
>> I think I've got analog properly installed and reasonably configured (I
>> haven't done "many" changes in the cfg file), but seem to be stalling
>> when I try to run it with my logs.
>>
>> It runs fine with the data from a single day but with a request for
>> results for 10 days either stalls or is running extremely slowly.
>>
>> I've tried changing the HOSTLOWMEM command and monitoring the progress
>> with PROGRESSFREQ (after 460 000 lines read it slows way down and seems
>> to stall after 780 000 lines). I have few other processes running
>> concurrently. Another question...does using the cache help in any way
>> with the speed of processing the logs?
>>
>> After consulting the archives of your "help" letters, I thought I found
>> a solution with regard to RAM/SWAP questions. However, my colleague had
>>
>> already tried to increase the SWAP but more and more memory is seemingly
>>
>> required...does the required RAM memory increase proportionally
>> linearly or exponentially?
>>
>
>The RAM required increases more slowly than linearly.
>
>You should be able to cope with a lot more lines than that unless another
>process is already using all the RAM. What OS are you on? You could use top
>(Unix) or the Task Manager (Windows NT) to examine the memory being used.
>
>Another thought. Have you got a lot of different filenames (for example, if
>they're dynamically generated?). If so, maybe you need FILELOWMEM instead of
>HOSTLOWMEM.
>
>--
>Stephen Turner [EMAIL PROTECTED] http://www.statslab.cam.ac.uk/~sret1/
> Statistical Laboratory, 16 Mill Lane, Cambridge CB2 1SB, England
> "Due to the conflict in Kosovo, we will not be showing the movie Wag the
> Dog. Instead, we will show Mortal Kombat: Annihilation." Cable & Wireless
>
>------------------------------------------------------------------------
>This is the analog-help mailing list. To unsubscribe from this
>mailing list, send mail to [EMAIL PROTECTED]
>with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
>List archived at http://www.mail-archive.com/[email protected]/
>------------------------------------------------------------------------
--
Alejandro Fernandez,
[EMAIL PROTECTED]
++44 0117 941 2284
http://charlotte.sift.co.uk/~ale/
35 Sandbed Road, St Werburghs, Bristol BS2 9TY
http://bedford.gcal.ac.uk/fernandeza/index.html
------------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
List archived at http://www.mail-archive.com/[email protected]/
------------------------------------------------------------------------
