I think understand the hole I've dug myself into-- but let me run this by the experts.
I have large log files (~1.5 gb each month, roll over monthly). I have lots of different organizations and applications that would like to have stats for just their little demesne (16 now, including a generic all inclusive one). Each report is slightly different, but not hugely. I was running all the reports one after the other, each with one main cfg file and then their subfiles. It runs in under 20 minutes on the 11th of a month... I don't recall how long it took on the 31st, but it was a bit longer. :) So I thought I would run the everything report first and create a nightly.cache file for the subsequent 15 app/site reports. It greatly speeds up the report making process-- but now the user report for each app is narfed up and shows users that don't use that site or application. I'm assuming that that is because the cache file strips out some of that data and the users data and request data tables are linked together anymore. Is my interpretation of events/problem correct? Is there a better way to speed this up and provide such application/site detail with Analog? Here's a sample of my command line(s): Executing the following: "TZ=CDT /tools/analog/current/analog -G -C'DEBUG ON' -C'WARNINGS ON' -C'CASE INSENSITIVE' -C'HOST ON' -C'DOMAIN Off' -C'HOSTEXCLUDE 10.1.3.' -C'DIRECTORY ON' -C'DIRFLOOR 10r' -C'DIRSORTBY REQUESTS' -C'GENSUMLINES BCEFGHIJMND' -C'PAGEINCLUDE *.cfm' -C'PAGEINCLUDE *.php' -C'PAGEINCLUDE *.cgi' -C'PAGEINCLUDE *.pl' -C'PAGEINCLUDE *.shtml' -C'PAGEINCLUDE *.asp' -C'PAGEINCLUDE *.pdf' -C'REQUEST On' -C'REQARGSFLOOR 5r' -C'REQCOLS NRD' -C'REQSORTBY REQUESTS' -C'FAILURE ON' -C'FAILCOLS rd' -C'FAILSORTBY REQUESTS' -C'REFERER Off' -C'REFSITE ON' -C'USER ON' -C'USERCOLS dbBR' -C'USERSORTBY REQUESTS' -C'ALLGRAPH P' -C'BARSTYLE E' -C'SUBDIRFLOOR 10r' -C'SUBDIRSORTBY REQUESTS' -C'FILEALIAS /org1/apt1/* /$1' -C'FILEALIAS /nwo/* /$1' -C'FILEALIAS */asdb/* /asdb/' -C'REQALIAS */asdb/* /asdb/' -C'FILEALIAS */osmos/* /osmo/' -C'REQALIAS */osmos/* /osmo/' -C'FILEALIAS */dkc/* /dkc/' -C'REQALIAS */dkc/* /dkc/' -C'FILEALIAS */dms/* /dms/' -C'REQALIAS */dms/* /dms/' -C'FILEALIAS */hrs* /hrs/' -C'REQALIAS */hrs* /hrs/' -C'FILEALIAS */oms* /oms/' -C'REQALIAS */oms* /oms/' -C'FILEALIAS */rma* /rma/' -C'REQALIAS */rma* /rma/' -C'FILEALIAS */it/* /it/' -C'REQALIAS */it/* /it/' -C'FILEALIAS */ldb/* /ldb/' -C'REQALIAS */ldb/* /ldb/' -C'FILEALIAS */midas* /midas/' -C'REQALIAS */midas* /midas/' -C'FILEALIAS */aca* /aca/' -C'REQALIAS */aca* /aca/' -C'FILEALIAS */adb/* /adb/' -C'REQALIAS */adb/* /adb/' -C'FILEALIAS */sans/* /sms/' -C'FILEALIAS */sms/* /sms/' -C'REQALIAS */sms/* /sms/' -C'FILEALIAS */ut/* /ut/' -C'REQALIAS */ut/* /ut/' -C'FILEALIAS */fi/* /FI/' -C'REQALIAS */fi/* /FI/' -C'FILEALIAS REGEXP:(vmd*) /vmd/' -C'REQALIAS REGEXP:(vmd*) /vmd/' -C'FILEALIAS REGEXP:(vmdhtml) /vmd/' -C'REQALIAS REGEXP:(vmdhtml) /vmd/' -C'FILEALIAS REGEXP:(asdb) /asdb/' -C'HOSTNAME www-d' -C'HOSTURL http://www-d############' -C'LANGUAGE US-ENGLISH' -C'LOGFORMAT (%S - %u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b %T)' -C'LOGFORMAT (%S - %j.%u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b %T)' -C'LOGFORMAT (%S -.%u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b %T)' -C'LOGFORMAT (%S %u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b %T)' -C'LOGFORMAT (%S %u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b %T %j)' -C'LOGFORMAT (%S - %u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b %T %j)' -C'LOGFORMAT (%S - %j.%u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b %T %j)' -C'LOGFORMAT (- - %u [%d/%M/%Y:%h:%n:%j] "%j %r %j" %c %b %T)' -C'LOGFORMAT (%Y %M %W%d %h:%n:%j] %S %u \\"%j %r %j\\" %c %b %T \\"%f\\" \\"%B\\")' -C'LOGFORMAT (%Y %M %W%d %h:%n:%j] %S %u "%j %r %j" %c %b %T "%f" "%B")' -C'LOGFORMAT (%Y %M %W%d %h:%n:%j] %S %u \\"%j %r %j\\" %c %b %t \\"%f\\" \\"%B\\")' -C'LOGFORMAT (%Y %M %W%d %h:%n:%j] %S %u \\"%j %r)' -C'LOGFORMAT (%Y %M %W%d %h:%n:%j %j %j %S %u \\"%j %r %j\\" %c %b %T \\"%f\\" \\"%B\\")' -C'LOGFORMAT (%Y %M %W%d %h:%n:%j %j-%j-%j %j:%j:%j %j %r %q %j %S %A %f %c %b %T)' -C'IMAGEDIR /systems/webreports/images/' -C'MONTHLY OFF' -C'YEARLY OFF' -C'LOGFILE /weblogs/apache/2007/09/combined.200709.log' -C'OUTFILE /www/systems/webreports/2007/09/_simple/index.html' -C'CACHEOUTFILE /tools/analog/current/cache/nightly.cache' 2>/tools/analog/current/cron.log" Apps number 1 of 15 is "/tools/analog/current/configs/asdb.cfg". Will now verify the output directory. I will check out this directory "/www/systems/webreports/2007/09/asdb". Directory exists-- you may continue. Executing the following: "TZ=CDT /tools/analog/current/analog +g/tools/analog/current/analog.cfg +g/tools/analog/current/configs/asdb.cfg" /tools/analog/current/analog: analog version 6.0/Unix /tools/analog/current/analog: Warning M: Logfile /weblogs/apache/2007/09/combined.200709.log contains lines with no hosts, which are being filtered (For help on all errors and warnings, see docs/errors.html) /tools/analog/current/analog: Warning M: Logfile /weblogs/apache/2007/09/combined.200709.log contains lines with no bytes: byte counts may be low F: Opening /tools/analog/6_0_nucache/lang/us.lng as language file F: Closing language file /tools/analog/6_0_nucache/lang/us.lng F: Opening /tools/analog/6_0_nucache/lang/usdom.tab as domains file F: Closing domains file /tools/analog/6_0_nucache/lang/usdom.tab F: Opening /tools/analog/6_0_nucache/lang/usdesc.txt as report descriptions file F: Closing report descriptions file /tools/analog/6_0_nucache/lang/usdesc.txt F: Opening /weblogs/apache/2007/09/combined.200709.log as logfile F: Closing logfile /weblogs/apache/2007/09/combined.200709.log S: Successful requests: 372100 S: Redirected requests: 2765 S: Failed requests: 3326 S: Requests returning informational status code: 0 S: Status code not given: 0 S: Unwanted lines: 0 S: Corrupt lines: 35 S: Earliest entry in logfile: 1/Sep/07:1053 S: Latest entry in logfile: 11/Sep/07:0912 F: Opening /tools/analog/current/cache/nightly.cache as cache output file F: Closing /tools/analog/current/cache/nightly.cache F: Opening /www/systems/webreports/2007/09/_simple/index.html as output file /tools/analog/current/analog: Warning R: Turning off empty Search Word Report F: Opening /www/systems/webreports/2007/09/_simple/org.png as pie chart file F: Closing /www/systems/webreports/2007/09/_simple/org.png F: Opening /www/systems/webreports/2007/09/_simple/host.png as pie chart file F: Closing /www/systems/webreports/2007/09/_simple/host.png F: Opening /www/systems/webreports/2007/09/_simple/user.png as pie chart file F: Closing /www/systems/webreports/2007/09/_simple/user.png F: Opening /www/systems/webreports/2007/09/_simple/refsite.png as pie chart file F: Closing /www/systems/webreports/2007/09/_simple/refsite.png F: Opening /www/systems/webreports/2007/09/_simple/os.png as pie chart file F: Closing /www/systems/webreports/2007/09/_simple/os.png F: Opening /www/systems/webreports/2007/09/_simple/code.png as pie chart file F: Closing /www/systems/webreports/2007/09/_simple/code.png F: Opening /www/systems/webreports/2007/09/_simple/size.png as pie chart file F: Closing /www/systems/webreports/2007/09/_simple/size.png F: Opening /www/systems/webreports/2007/09/_simple/type.png as pie chart file F: Closing /www/systems/webreports/2007/09/_simple/type.png F: Opening /www/systems/webreports/2007/09/_simple/dir.png as pie chart file F: Closing /www/systems/webreports/2007/09/_simple/dir.png F: Opening /www/systems/webreports/2007/09/_simple/fail.png as pie chart file F: Closing /www/systems/webreports/2007/09/_simple/fail.png F: Opening /www/systems/webreports/2007/09/_simple/req.png as pie chart file F: Closing /www/systems/webreports/2007/09/_simple/req.png F: Closing /www/systems/webreports/2007/09/_simple/index.html /tools/analog/current/analog: analog version 6.0/Unix F: Closing configuration file /tools/analog/6_0_nucache/analog.cfg /tools/analog/current/analog: Warning D: LOGFORMAT in configuration file analog.cfg with no subsequent LOGFILE (For help on all errors and warnings, see docs/errors.html) F: Opening /tools/analog/current/analog.cfg as configuration file F: Closing configuration file /tools/analog/current/analog.cfg /tools/analog/current/analog: Warning D: LOGFORMAT in configuration file /tools/analog/current/analog.cfg with no subsequent LOGFILE F: Opening /tools/analog/current/configs/asdb.cfg as configuration file F: Closing configuration file /tools/analog/current/configs/asdb.cfg F: Opening /tools/analog/6_0_nucache/lang/us.lng as language file F: Closing language file /tools/analog/6_0_nucache/lang/us.lng F: Opening /tools/analog/6_0_nucache/lang/usdom.tab as domains file F: Closing domains file /tools/analog/6_0_nucache/lang/usdom.tab F: Opening /tools/analog/6_0_nucache/lang/usdesc.txt as report descriptions file F: Closing report descriptions file /tools/analog/6_0_nucache/lang/usdesc.txt F: Opening /tools/analog/current/cache/nightly.cache as cache file /tools/analog/current/analog: Warning M: Cache file /tools/analog/current/cache/nightly.cache doesn't contain last-seven-day statistics; so these may be underestimated F: Closing cache file /tools/analog/current/cache/nightly.cache S: Successful requests: 372100 S: Redirected requests: 2765 S: Failed requests: 3326 S: Requests returning informational status code: 0 S: Status code not given: 0 S: Earliest entry in cache file: 1/Sep/07:1053 S: Latest entry in cache file: 11/Sep/07:0912 F: Opening /tools/analog/current/cache/asdb as cache file /tools/analog/current/analog: Warning M: Cache file /tools/analog/current/cache/asdb doesn't contain last-seven-day statistics; so these may be underestimated F: Closing cache file /tools/analog/current/cache/asdb S: Successful requests: 586659 S: Redirected requests: 24063 S: Failed requests: 36102 S: Requests returning informational status code: 0 S: Status code not given: 0 S: Earliest entry in cache file: 3/Oct/06:0954 S: Latest entry in cache file: 10/Sep/07:0827 /tools/analog/current/analog: Warning L: cache files /tools/analog/current/cache/nightly.cache and /tools/analog/current/cache/asdb overlap: possible double counting F: Opening /www/systems/webreports/2007/09/asdb/index.html as output file F: Opening /www/systems/webreports/2007/09/asdb/user.png as pie chart file F: Closing /www/systems/webreports/2007/09/asdb/user.png F: Opening /www/systems/webreports/2007/09/asdb/dir.png as pie chart file F: Closing /www/systems/webreports/2007/09/asdb/dir.png F: Opening /www/systems/webreports/2007/09/asdb/fail.png as pie chart file F: Closing //www/systems/webreports/2007/09/asdb/fail.png F: Closing /www/systems/webreports/2007/09/asdb/index.html Thanks. -- -Kevin Admit your errors before someone else exaggerates them. - Andrew V. Mason aych tee tee pee colon slash slash texascreasons dot homedns dot org
+------------------------------------------------------------------------ | TO UNSUBSCRIBE from this list: | http://lists.meer.net/mailman/listinfo/analog-help | | Analog Documentation: http://analog.cx/docs/Readme.html | List archives: http://www.analog.cx/docs/mailing.html#listarchives | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general +------------------------------------------------------------------------
