Happy day before the 4th everyone,
I've been dealing with some netscape log files this weekend that refuse
to be processed by analog. I am including the log format from the
access_logs, the LOGFORMAT line from the stats config, a sample line of
the access log, the command line I am using the run analog. This is
Netscape 4 with Analog 6.0. Analog is simply pumping out 99kb cache
files and empty stats files. Sometimes it's almost as if it's not even
seeing the data being piped to it. Could the log format line of the
access logs be corrupting this somehow? Or does anyone see what is
happening and/or how to fix it? Thanks alot in advance for your time.
Note also that I have run analog without specifying a LOGFORMAT line so
that it can actively create a LOGFORMAT line but this returns the same
empty results.
*Log format from the access logs...*
format=%Ses->client.ip% - %Req->vars.auth-user% [%SYSDATE%]
"%Req->reqpb.clf-request%" %Req->srvhdrs.clf-status%
%Req->srvhdrs.content-length% "%Req->headers.referer%"
"%Req->headers.user-agent%" %Req->headers.host%
*LOGFORMAT from stats.conf...*
LOGFORMAT (%S - %u [%d/%M/%Y:%h:%n:%j %j] "%j %r %j" %c %b "%f "%B" %v)
*Sample access log entry...*
38.113.234.180 - - [01/Jun/2006:00:02:18 -0400] "GET
/stories/021904/opi_20040219017.shtml HTTP/1.0" 200 35937 "-"
"voyager/1.0" www.whatever.com
*Here is what I am running to process the access_logs...*
for FILE in `ls /path/ | grep access_log`
do
DATE=`echo ${FILE} | awk -F. '{print $2}'`
grep -v \/RealMedia\/ /path/access_log.${DATE} | /usr/bin/analog
+gstats.conf +O/path/${DATE}.html +C"HOSTNAME \"${SITE}\"" +C"HOSTURL
http://${SITE}"; +C"CACHEOUTFILE /path/cache.${DATE}" 2>>analog.log
done
/usr/bin/analog +gstats.conf +C"LOGFILE NONE" +C"HOSTNAME \"${SITE}\""
+C"HOSTURL http://${SITE}"; +U/path/cache.*
+O/path/${MONTH}${YEAR}composite.html 2>>analog.log
*And here is part of the analog.log...*
/usr/bin/analog: analog version 6.0/Unix
/usr/bin/analog: Warning D: LOGFORMAT in configuration file stats.conf
with no
subsequent LOGFILE
(For help on all errors and warnings, see docs/errors.html)
/usr/bin/analog: Warning D: In File Type Report, FLOOR (requests) doesn't
match SUBFLOOR (bytes)
/usr/bin/analog: Warning D: In File Type Report, SORTBY (bytes) doesn't
match
FLOOR (requests)
/usr/bin/analog: Warning D: In Directory Report, SORTBY (requests) doesn't
match SUBSORTBY (bytes)
/usr/bin/analog: Warning D: In Directory Report, FLOOR (requests) doesn't
match SUBFLOOR (bytes)
/usr/bin/analog: Warning D: In Directory Report, SUBSORTBY (bytes) isn't
included in COLS
/usr/bin/analog: Warning D: In Directory Report, SUBFLOOR (bytes) isn't
included in COLS
/usr/bin/analog: Warning F: Failed to open logfile /usr/bin/logfile.log:
ignoring it
/usr/bin/analog: Warning R: Turning off empty time reports
/usr/bin/analog: Warning R: Turning off empty Request Report
/usr/bin/analog: Warning R: Turning off empty File Type Report
/usr/bin/analog: Warning R: Turning off empty Directory Report
/usr/bin/analog: Warning R: Turning off empty Host Report
/usr/bin/analog: Warning R: Turning off empty Domain Report
/usr/bin/analog: Warning R: Turning off empty Organisation Report
/usr/bin/analog: Warning R: Turning off empty Search Query Report
/usr/bin/analog: Warning R: Turning off empty Search Word Report
/usr/bin/analog: Warning R: Turning off empty Internal Search Query Report
/usr/bin/analog: Warning R: Turning off empty Operating System Report
/usr/bin/analog: Warning R: Turning off empty File Size Report
/usr/bin/analog: Warning R: Turning off empty Status Code Report
/usr/bin/analog: analog version 6.0/Unix
/usr/bin/analog: Warning D: LOGFORMAT in configuration file stats.conf
with no
subsequent LOGFILE
(For help on all errors and warnings, see docs/errors.html)
/usr/bin/analog: Warning D: In File Type Report, FLOOR (requests) doesn't
match SUBFLOOR (bytes)
/usr/bin/analog: Warning D: In File Type Report, SORTBY (bytes) doesn't
match
FLOOR (requests)
/usr/bin/analog: Warning D: In Directory Report, SORTBY (requests) doesn't
match SUBSORTBY (bytes)
/usr/bin/analog: Warning D: In Directory Report, FLOOR (requests) doesn't
match SUBFLOOR (bytes)
/usr/bin/analog: Warning D: In Directory Report, SUBSORTBY (bytes) isn't
included in COLS
/usr/bin/analog: Warning D: In Directory Report, SUBFLOOR (bytes) isn't
included in COLS
/usr/bin/analog: Warning R: Turning off empty time reports
/usr/bin/analog: Warning R: Turning off empty Request Report
/usr/bin/analog: Warning R: Turning off empty File Type Report
/usr/bin/analog: Warning R: Turning off empty Directory Report
/usr/bin/analog: Warning R: Turning off empty Host Report
/usr/bin/analog: Warning R: Turning off empty Domain Report
/usr/bin/analog: Warning R: Turning off empty Organisation Report
/usr/bin/analog: Warning R: Turning off empty Search Query Report
/usr/bin/analog: Warning R: Turning off empty Search Word Report
/usr/bin/analog: Warning R: Turning off empty Internal Search Query Report
/usr/bin/analog: Warning R: Turning off empty Operating System Report
/usr/bin/analog: Warning R: Turning off empty File Size Report
/usr/bin/analog: Warning R: Turning off empty Status Code Report
*And of course my stats config file.*
HOSTNAME "whatever.com"
HOSTURL http://whatever.com
FILEINCLUDE /cgi-bin/*/cqcgi/@*.env*
PAGEINCLUDE /cgi-bin/*/cqcgi/@*.env*
WARNINGS ON
DEBUG OFF
##############################################################################
# ANALOG_STATS.CONF
##############################################################################
### FOR QUICK REFERENCE TO COMMANDS IN THIS FILE SEE
http://www.statslab.cam.ac.uk/~sret1/analog/docs/quickref.html#quicklang ###
###############################################
LOGFORMAT (%S - %u [%d/%M/%Y:%h:%n:%j %j] "%j %r %j" %c %b "%f "%B" %v)
## Where the images that will be displayed on the statistic report are
located.
#IMAGEDIR /images/analog/
## What to include in the request report.
REQINCLUDE PAGES
## Used for linking to what the reports are showing. (this one says link
to what is being represented as a "page". [determined by REQINCLUDE]).
LINKINCLUDE PAGES
## What order the reports should be displayed in.
REPORTORDER x1QmWDdHwh4657oZSlLujJkKfsNnBbpvRMcPztiEIYyr
## Which files are included and excluded from the report. (FILEEXCLUDE
Overrides FILEINCLUDE).
FILEINCLUDE /*
FILEEXCLUDE *.gif,*.jpg
FILEEXCLUDE /RealMedia/*
## What type of file to count as a "page".
PAGEINCLUDE *.shtml
PAGEINCLUDE *.html
PAGEINCLUDE *.htm
PAGEEXCLUDE default.ida
## What hosts will be excluded from the analog statistics report.
HOSTEXCLUDE 209.120.215.*
HOSTEXCLUDE 204.155.172.9
HOSTEXCLUDE 204.155.171.19
HOSTEXCLUDE 216.39.48.62
HOSTEXCLUDE 216.39.48.121
HOSTEXCLUDE 216.239.46.*
HOSTEXCLUDE 217.204.41.131
HOSTEXCLUDE 209.73.164.*
HOSTEXCLUDE 204.162.96.*
HOSTEXCLUDE 204.162.98.*
HOSTEXCLUDE 208.219.77.*
HOSTEXCLUDE 209.157.29.*
HOSTEXCLUDE 209.202.193.*
HOSTEXCLUDE 64.208.37.*
HOSTEXCLUDE 171.78.41.185
HOSTEXCLUDE 198.3.103.105
HOSTEXCLUDE 198.3.103.108
HOSTEXCLUDE 198.3.103.112
HOSTEXCLUDE 198.3.103.24
HOSTEXCLUDE 198.3.103.35
HOSTEXCLUDE 198.3.103.50
HOSTEXCLUDE 198.3.103.56
HOSTEXCLUDE 198.3.103.57
HOSTEXCLUDE 198.3.103.58
HOSTEXCLUDE 198.3.103.59
HOSTEXCLUDE 198.3.103.60
HOSTEXCLUDE 198.3.103.65
HOSTEXCLUDE 198.3.103.66
HOSTEXCLUDE 198.3.103.68
HOSTEXCLUDE 198.3.103.69
HOSTEXCLUDE 198.3.103.70
HOSTEXCLUDE 198.3.103.72
HOSTEXCLUDE 198.3.103.81
HOSTEXCLUDE 198.3.103.84
HOSTEXCLUDE 198.3.103.93
HOSTEXCLUDE 198.3.103.97
HOSTEXCLUDE 198.5.208.100
HOSTEXCLUDE 198.5.210.181
HOSTEXCLUDE 198.5.210.189
HOSTEXCLUDE 199.172.148.105
HOSTEXCLUDE 199.172.149.131
HOSTEXCLUDE 199.172.149.132
HOSTEXCLUDE 199.172.149.138
HOSTEXCLUDE 199.172.149.139
HOSTEXCLUDE 199.172.149.140
HOSTEXCLUDE 199.172.149.141
HOSTEXCLUDE 199.172.149.142
HOSTEXCLUDE 199.172.149.143
HOSTEXCLUDE 199.172.149.144
HOSTEXCLUDE 199.172.149.161
HOSTEXCLUDE 199.172.149.165
HOSTEXCLUDE 199.172.152.54
HOSTEXCLUDE 199.172.156.172
HOSTEXCLUDE 199.172.157.28
HOSTEXCLUDE 202.212.5.30
HOSTEXCLUDE 202.212.5.36
HOSTEXCLUDE 202.9.155.123
HOSTEXCLUDE 203.147.31.123
HOSTEXCLUDE 204.162.97.1
HOSTEXCLUDE 204.162.97.17
HOSTEXCLUDE 204.162.97.231
HOSTEXCLUDE 204.62.245.167
HOSTEXCLUDE 204.62.245.178
HOSTEXCLUDE 204.62.245.187
HOSTEXCLUDE 204.62.245.32
HOSTEXCLUDE 204.123.28.10
HOSTEXCLUDE 204.123.28.11
HOSTEXCLUDE 205.181.75.65
HOSTEXCLUDE 205.181.75.66
HOSTEXCLUDE 205.216.162.70
HOSTEXCLUDE 205.226.201.30
HOSTEXCLUDE 205.226.203.35
HOSTEXCLUDE 205.226.203.56
HOSTEXCLUDE 205.226.204.238
HOSTEXCLUDE 205.253.141.100
HOSTEXCLUDE 206.58.218.16
HOSTEXCLUDE 206.79.171.156
HOSTEXCLUDE 206.79.171.157
HOSTEXCLUDE 206.79.171.16
HOSTEXCLUDE 206.79.171.17
HOSTEXCLUDE 206.79.171.18
HOSTEXCLUDE 206.79.171.19
HOSTEXCLUDE 206.79.171.20
HOSTEXCLUDE 206.79.171.21
HOSTEXCLUDE 206.79.171.22
HOSTEXCLUDE 206.79.171.23
HOSTEXCLUDE 206.79.171.24
HOSTEXCLUDE 206.79.171.25
HOSTEXCLUDE 206.79.171.67
HOSTEXCLUDE 206.79.171.81
HOSTEXCLUDE 206.79.171.85
HOSTEXCLUDE 206.79.171.89
HOSTEXCLUDE 206.79.171.93
HOSTEXCLUDE 206.79.171.97
HOSTEXCLUDE 207.138.42.105
HOSTEXCLUDE 207.176.248.16
HOSTEXCLUDE 207.239.241.250
HOSTEXCLUDE 207.77.90.136
HOSTEXCLUDE 207.77.90.14
HOSTEXCLUDE 207.77.90.183
HOSTEXCLUDE 207.77.90.185
HOSTEXCLUDE 207.77.90.186
HOSTEXCLUDE 207.77.90.187
HOSTEXCLUDE 207.77.90.20
HOSTEXCLUDE 207.77.91.184
HOSTEXCLUDE 207.88.29.130
HOSTEXCLUDE 208.146.26.19
HOSTEXCLUDE 208.146.26.233
HOSTEXCLUDE 208.146.27.123
HOSTEXCLUDE 208.146.27.124
HOSTEXCLUDE 208.146.27.94
HOSTEXCLUDE 208.146.27.95
HOSTEXCLUDE 208.146.27.96
HOSTEXCLUDE 208.184.224.199
HOSTEXCLUDE 208.51.0.79
HOSTEXCLUDE 209.1.12.68
HOSTEXCLUDE 209.1.13.231
HOSTEXCLUDE 209.1.13.232
HOSTEXCLUDE 209.131.48.100
HOSTEXCLUDE 209.131.48.101
HOSTEXCLUDE 209.131.48.78
HOSTEXCLUDE 209.97.173.98
HOSTEXCLUDE 209.97.173.96
HOSTEXCLUDE 209.185.143.138
HOSTEXCLUDE 209.202.192.141
HOSTEXCLUDE 209.67.228.106
HOSTEXCLUDE 209.67.228.107
HOSTEXCLUDE 209.67.228.109
HOSTEXCLUDE 209.67.228.15
HOSTEXCLUDE 209.67.228.154
HOSTEXCLUDE 209.67.228.156
HOSTEXCLUDE 209.67.228.159
HOSTEXCLUDE 209.67.228.166
HOSTEXCLUDE 209.67.228.17
HOSTEXCLUDE 209.67.228.170
HOSTEXCLUDE 209.67.228.174
HOSTEXCLUDE 209.67.228.178
HOSTEXCLUDE 209.67.228.182
HOSTEXCLUDE 209.67.228.19
HOSTEXCLUDE 209.67.228.21
HOSTEXCLUDE 209.67.228.23
HOSTEXCLUDE 209.67.228.42
HOSTEXCLUDE 209.73.79.29
HOSTEXCLUDE 209.73.228.160
HOSTEXCLUDE 209.73.228.161
HOSTEXCLUDE 209.73.228.162
HOSTEXCLUDE 209.73.228.163
HOSTEXCLUDE 209.73.228.164
HOSTEXCLUDE 209.73.228.165
HOSTEXCLUDE 209.73.228.166
HOSTEXCLUDE 209.73.228.167
HOSTEXCLUDE 209.73.228.168
HOSTEXCLUDE 209.73.228.169
HOSTEXCLUDE 209.73.228.170
HOSTEXCLUDE 209.73.228.171
HOSTEXCLUDE 209.73.228.172
HOSTEXCLUDE 209.73.228.173
HOSTEXCLUDE 209.73.228.174
HOSTEXCLUDE 209.73.228.175
HOSTEXCLUDE 209.249.67.103
HOSTEXCLUDE 210.236.233.155
HOSTEXCLUDE 212.111.41.153
HOSTEXCLUDE 212.111.41.33
HOSTEXCLUDE 212.111.41.34
HOSTEXCLUDE 212.111.41.35
HOSTEXCLUDE 212.111.41.36
HOSTEXCLUDE 212.187.226.111
HOSTEXCLUDE 212.209.54.134
HOSTEXCLUDE 213.189.83.100
HOSTEXCLUDE 216.0.86.126
HOSTEXCLUDE 216.0.86.75
HOSTEXCLUDE 216.167.107.159
HOSTEXCLUDE 216.251.230.194
HOSTEXCLUDE 216.251.230.178
HOSTEXCLUDE 216.32.237.11
HOSTEXCLUDE 216.32.237.12
HOSTEXCLUDE 216.32.237.13
HOSTEXCLUDE 216.32.237.14
HOSTEXCLUDE 216.32.237.16
HOSTEXCLUDE 216.32.237.17
HOSTEXCLUDE 216.32.237.18
HOSTEXCLUDE 216.32.237.19
HOSTEXCLUDE 216.32.237.20
HOSTEXCLUDE 216.32.237.21
HOSTEXCLUDE 216.32.237.22
HOSTEXCLUDE 216.32.237.23
HOSTEXCLUDE 216.32.237.24
HOSTEXCLUDE 216.32.237.25
HOSTEXCLUDE 216.32.237.27
HOSTEXCLUDE 216.32.237.28
HOSTEXCLUDE 216.32.237.29
HOSTEXCLUDE 216.32.237.7
HOSTEXCLUDE 216.34.109.190
HOSTEXCLUDE 216.34.109.191
HOSTEXCLUDE 216.35.194.183
HOSTEXCLUDE 216.35.194.184
HOSTEXCLUDE 216.35.194.185
HOSTEXCLUDE 216.35.194.186
HOSTEXCLUDE 216.35.194.187
HOSTEXCLUDE 216.35.194.188
HOSTEXCLUDE 216.35.194.189
HOSTEXCLUDE 216.35.194.190
HOSTEXCLUDE 64.14.36.46
HOSTEXCLUDE 64.71.132.198
HOSTEXCLUDE 64.68.86.* #googlebot
HOSTEXCLUDE 64.68.87.* #googlebot
HOSTEXCLUDE 64.68.88.* #googlebot
HOSTEXCLUDE 64.208.33.33
HOSTEXCLUDE 64.208.37.42
HOSTEXCLUDE 64.208.37.54
HOSTEXCLUDE 64.208.37.77
HOSTEXCLUDE 64.37.114.34
###### teoma ips - nrg changed to ip blocks 091304 #########
HOSTEXCLUDE 65.214.36.*
HOSTEXCLUDE 65.214.37.*
HOSTEXCLUDE 65.214.38.*
HOSTEXCLUDE 65.214.39.*
############################
HOSTEXCLUDE 66.77.73.58
HOSTEXCLUDE 66.77.73.64
HOSTEXCLUDE 66.77.73.65
HOSTEXCLUDE 66.77.73.79
HOSTEXCLUDE 66.77.73.140
HOSTEXCLUDE 66.77.73.146
HOSTEXCLUDE 66.77.73.147
HOSTEXCLUDE 66.77.73.155
HOSTEXCLUDE 66.77.73.183
HOSTEXCLUDE 66.77.73.185
HOSTEXCLUDE 66.77.73.245
HOSTEXCLUDE 66.77.73.246
## Reports that will be displayed. (either ON or OFF)
GENERAL ON #General Summary
MONTHLY ON #Monthly Report
WEEKLY ON #Weekly Report
FULLDAILY ON #Daily Report
DAILY ON #Daily Summary
HOURLY ON #Hourly Summary
HOST ON #Host Report
REQUEST ON #Request Report
DIRECTORY ON #Directory Report
INTSEARCHQUERY ON #Internal Search Query Report
SEARCHQUERY ON #Search Engine Query Report
## How Analog knows what day to start a weekly report.
#WEEKBEGINSON SUNDAY
## What the Reports will be sorted by.
HOSTSORTBY REQUESTS
DIRSORTBY REQUESTS
VHOSTSORTBY REQUESTS
REFSORTBY REQUESTS
## Lowest amount of {...} that the report will report on.
#VHOSTFLOOR 0r
DIRFLOOR -50r
REFFLOOR 200r
REFARGSFLOOR 100r
TYPEFLOOR -10r
REDIRFLOOR 500r
REQFLOOR 200r
INTSEARCHQUERYFLOOR 1r
SEARCHQUERYFLOOR 1r
## What type of output to serve the report in.
#OUTPUT HTML
#OUTPUT PREFORMATTED
#PRESEP :
## Search Engine Config
INTSEARCHENGINE /cgi-bin/*/cqcgi/@*.env
CQ_QUERY_STRING,CQ_QUERY_STRING.mcc_pub_date,CQ_QUERY_STRING.mcc_byline1
SEARCHENGINE http://*google*/* q
SEARCHENGINE http://*yahoo*/* p
SEARCHENGINE http://*msn*/* q
##DNS DATA
#DNSFILE /var/opt/analog/dns.cache #Tells where DNS address
translation is kept.
#DNSLOCKFILE /var/tmp/analogdns.lock #Tells where the lock file is when
analog runs.
#DNSGOODHOURS 720 #How long before looking the
number up again.
#DNS WRITE #What method to use when looking
dns up.
## Debugs and warnings.
WARNINGS ON
DEBUG OFF
DIRCOLS R
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
