Quoting "Ulf Hofemeier" <[EMAIL PROTECTED]>:
Quoting Aengus <[EMAIL PROTECTED]>:
Ulf Hofemeier <[EMAIL PROTECTED]> wrote:
I have to admit that the analog configuration is getting quite
confusing to me. Unfortunately the APACHELOGFORMAT line doesn't solve
my problem, so please allow me to provide you with a little more
information regarding the purpose of the updated analog.cfg, as well
as what I'm doing before the problem occurs.
1. Copy the previous month Apache log to a temporary location
2. Run a script to extract page visitor data from the general Apache
log file and store it in a separate file
3. Run a bash 'for i' loop on the new log files and store the data in
page visitor sub directores
Unfortunately I decided that Apache has to write more information to
its access_log log file, which is finally the reason why there is
issues with analog now. According to the analog documentation there is
a way to set up a hierarchy so that it will understand a log file
syntax even if it changes from old to new over time, but I haven't
been able to figure out how to make it work.
If you have multiple LOGFORMAT statements, Analog will try them
each in turn until it finds one that matches the entries in each of
your logiles. That means that if you have multiple logfiles, and
they aren't all the same format, Analog can still create a single
report from these different logfiles. Obviously the report may
understate this items that weren't recorded in some of the logfiles
- for example, you might have a million requests, but only only
200,000 Browser strings if you only added that field in leater log
files.
LOGFORMAT commands apply to LOGFILEs that are specified after the
LOGFORMAT in the .cfg file. DEFAULTLOGFORMAT commands apply to
logfiles that are specified on the command line.
It's not clear from your description whether your script calls
Analog and passes it the name of the logfile as a paramter, or
whether Analog picks up the logfile from the LOGFILE log-????-??.gz
statement in your .cfg file.
If you're speciying the LOGFILES in the .cfg file, then these lines
should do the job:
APACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b)
APACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\")
My script calls analog like this:
cd $datadir/$domain
# Determine the range of months from the list of log files that were
not empty.
first=`ls log*|sort|head -1|cut -b5-`
first=`echo $first |rev |cut -b4- |rev` ; # YEAR-MO
last=`ls access_log*|sort|tail -1|cut -b12-`
last=`echo $last |rev |cut -b4- |rev` ; # YEAR-MO
range="$first--$last"; # YEAR-MO--YEAR-MO
# Collect summary information from all the log files.
/data/stats/analog/analog access_log.????-??.gz >
$analogdir/$domain/$range.html
# Collect information by month in seperate files.
for i in access_log.????-??.gz ;
do
file=`echo $i |cut -b12-` # YEAR-MO.gz
file=`echo $file |rev |cut -b4- |rev` ; # YEAR-MO
/data/stats/analog/analog $i > $analogdir/$domain/$file.html
done
So I pass the log file to analog as a parameter on the command line
rather than using analog.cfg.
If you're calling Analog with the logfiles specified on the command
line, then these lines should work:
DEFAULTAPACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b)
DEFAULTAPACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b
\"%{Referer}i\" \"%{User-Agent}i\")
I will give these two lines a try in my analog.cfg.
Thank you.
Adding DEFAULTAPACHELOGFORMAT didn't work (or my analog.cfg is still broken)
analog.cfg looks like this:
# If you need a LOGFORMAT command (most people don't -- try it without
first!),
# it must go here, above the LOGFILE commands.
APACHELOGFORMAT (%S %l \"%u\" %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\")
DEFAULTLOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b)
DEFAULTLOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b)
DEFAULTLOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%j %r" %c %b)
DEFAULTLOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%r" %c %b)
DEFAULTAPACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b)
DEFAULTAPACHELOGFORMAT (%h %l \"%u\" %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\")
LOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b)
LOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%j %r" %c %b)
LOGFORMAT (%S %j "%u" [%d/%M/%Y:%h:%n:%j] "%r" %c %b)
LOGFORMAT (%S %l \"%u\" %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\")
DEBUG ON
LOGFILE log-????-??.gz
The output after running analog on the latest apache log file looks like this:
/data/stats/analog/analog -G +g../../analog.cfg access_log.2008-09.gz
>test.html
C: 1.2.3.4 - "-" [17/Sep/2008:13:12:52 -0600] "GET
/images/sandiamountains.jpg HTTP/1.1" 200 61671 "http://ladb.unm.edu/";
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13
(KHTML, like Gecko) Chrome/0.2.149.29 Safari/525.13"
C:
*
C: 1.2.3.4 - "-" [17/Sep/2008:13:12:53 -0600] "GET /favicon.ico
HTTP/1.1" 200 437 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US)
AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.2.149.29 Safari/525.13"
C:
*
C: 1.2.3.4 - "-" [27/Sep/2008:17:55:25 -0600] "GET / HTTP/1.1" 200
6576
"http://www.google.com/search?q=latin+america+databases&ie=utf-8&oe=utf-8&aq=t&rls=com.yahoo:en-US:official&client=firefox"; "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 YFF3
Firefox/3.0.1"
C:
*
F: Closing logfile access_log.2008-09.gz
S: Successful requests: 0
S: Redirected requests: 0
S: Failed requests: 0
S: Requests returning informational status code: 0
S: Status code not given: 0
S: Unwanted lines: 0
S: Corrupt lines: 23
/data/stats/analog/analog: Warning L: Large number of corrupt lines in logfile
access_log.2008-09.gz: turn debugging on or try different LOGFORMAT
Current logfile format:
%S %j "%j" [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b\n
%S %j "%j" [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b\n
%S %j "%j" [%d/%M/%Y:%h:%n:%j] "%j %r" %c %b\n
%S %j "%j" [%d/%M/%Y:%h:%n:%j] "%r" %c %b\n
F: Opening stdout as output file
F: Opening /data/stats/analog/requireanalogheader as header file
F: Closing header file /data/stats/analog/requireanalogheader
F: Opening /data/stats/analog/requireanalogfooter as footer file
F: Closing footer file /data/stats/analog/requireanalogfooter
+------------------------------------------------------------------------
| TO UNSUBSCRIBE from this list:
| http://lists.meer.net/mailman/listinfo/analog-help
|
| Analog Documentation: http://analog.cx/docs/Readme.html
| List archives: http://www.analog.cx/docs/mailing.html#listarchives
| Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
+------------------------------------------------------------------------
