[analog-help] large log files - long processing times

[amesclar]

Background -

I have a large log file (300+mb) on a Sun E250 Solaris server running v/4.11 of analog. This 300+mb file is created by cleaning up several smaller files (representing dates in the past) and combining the cleaned files. It takes the analog process approximately 5 hours to process the file. When I monitor system resources during execution using "top" I see about 45% cpu consumption and about 45% iowait consumption with the remaining 10% going to other misc. system stuff.

 

Initial plan -

Since the system is io bound, I was wondering if I could process these old log files once, since they contain static data, then combine these analog results with a set of analog results for the up-to-date stuff? After reviewing the documentation, it seems like the "cache" option was designed to accommodate this.

 

However, when I tried to implement the recommended test scenario, the initial plan failed.

 

Initial implementation -

Following the advice promoted in the online documentation for the "cache" option, I first ran analog against the log file which was about 20mb. The output file and cache file was created in about 15-20 seconds. I then wanted to create the analog results using the cache file so I could compare the outfile results. I killed the job after 2minutes without results. During the 2minutes, the process cpu utilization ranged from 40% at the beginning to 10% at the end just before killing the job.

 

Please note-

I have requested that our sys admin install the latest version of analog since I notice it has been improved, and that v/5.11 is now available.

 

Online areas reviewed -

1) cache documentation

2) searchable online help - searched for "large files"

3) how to ask questions to get answers

 

Questions -

1) Am I on the right track? That is, am I using the "cache" option the way it was intended?

2) If "cache" is the appropriate solution to my large file problem, how should I proceed in troubleshooting why the system seems to hang when processing the "cache" input?

3) If "cache" is not the appropriate solution to my large file problem, what other options should I investigate?

 

Thank you in advance.

 

Fred Ames
Wanda Application Development
Voice 415.477.7339, Pager 888.991.6253
"These opinions are strictly my own and not necessarily those of Wells Fargo."