-----Original Message----- From: Yuri N. Fominov Sent: Thursday, September 11, 2003 21:33 To: 'Kent, Mr. John' Subject: RE: [squid-users] Band New Squid Log Analyzer!
Thanks for feedback, Seems you are using HTTPD (Common Log Format) log emulation, unfortunately SCALAR can deal only with native Squid 2 and above log files... Regards, Yuri N. Fominov Corporate Solutions Manager ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ R.I.S.K. Company Baku, AZ-1022, Azerbaijan 59 Rashid Behbudov str. tel: +(99412) 973737 fax: +(99412) 981993 http://www.risk.az ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -----Original Message----- From: Kent, Mr. John [mailto:[EMAIL PROTECTED] Sent: Thursday, September 11, 2003 21:28 To: Yuri N. Fominov Subject: RE: [squid-users] Band New Squid Log Analyzer! Yuri, Thanks for making your code available. I tried it, but I don't think it worked properly, or more likely my logs are different than you expected. Here is the output of your program: bash-2.05$ ./scalar.awk /users/webuser/www_squid/logs/access.log ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~ ~~~~ SCALAR (Squid Cache Advanced Log Analyzer & Reporter), Version 0.91 ~~~~ ~~~~ ~~~~ ~~~~ (C) 2003 by Yuri N. Fominov, [EMAIL PROTECTED], http://scalar.risk.az ~~~~ ~~~~ SCALAR has no warranty and it is completely free, so you are welcome ~~~~ ~~~~ to re-distribute this pretty useful piece of software. Good Luck2All ~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~ ~~~ Analysis Headlines ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Log Start Time [-30-02-1970 09:03:14] Log End Time [-30-02-1970 09:01:08] Lines Analysed 85.395K Analysis Took: 10 sec. Average: 8539 Lines/sec Unique Hosts: 1 In Traffic: -59776500 B Out Traffic: -59776500 B ------------------------------------ Saved Traffic: 0 B -0.00 % ~~~ Requests By Hours ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 85.4K == 81.1K == 76.8K == 72.6K == 68.3K == 64.0K == 59.8K == 55.5K == 51.2K == 47.0K == 42.7K == 38.4K == 34.2K == 29.9K == 25.6K == 21.3K == 17.1K == 12.8K == 8.5K == 4.3K == ________________________________________________________________________ _______ Hours: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Maximum Requests: 85.395K/hour, 1.423K/min, 23.7/sec. Average Requests: 3.558K/hour, 59 /min, 1.0/sec. Minimum Requests: 0 /hour, 0 /min, 0.0/sec. ~~~ Traffic by Hours ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == 0B == == == == == == == == == == == == == == == == == == == == == == == ________________________________________________________________________ _______ Hours: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Maximum Xfers: 0 B/hour, 0 B/min, 0 B/sec [ 0 kbps] Average Xfers:-2490687 B/hour, -41511 B/min, -692 B/sec [ -5 kbps] Minimum Xfers:-59776500 B/hour, -996275 B/min, -16605 B/sec [-130 kbps] ~~~ File Extensions Report ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -------|------ R E Q U E S T S --------|----------- T R A F F I C -------|-----------| Ext. | total | misses | hits |hit% | total | misses | hits |hit%| -------|--------|--------|--------|-----|----------|----------|--------- -|----| ================|========|========|=====|==========|==========|========= =|====| TOTALS: 0 0 0 awk: ./scalar.awk:1084: (FILENAME=/users/webuser/www_squid/logs/access.log FNR=85395) fatal: division by zero attempted bash-2.05$ Here is a snippet of the log file I asked it to analyze: 15066,1 17% 206.48.59.87 - - [11/Sep/2003:03:32:14 -0700] "GET http://www.nrlmry.navy.mil:7777/java_scripts/jsClockGMT.js HTTP/1.1" 304 231 TCP_IMS_HIT:NONE 63.101.13.176 - - [11/Sep/2003:03:32:14 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/tc_thumbs/smal1320 03.03091106.gif HTTP/1.1" 200 1427 66 TCP_HIT:NONE 206.48.59.87 - - [11/Sep/2003:03:32:15 -0700] "GET http://www.nrlmry.navy.mil:7777/tc-bin/tc_home2.cgi?YEAR=2003&MO=Sep&BAS IN=ATL&STORM_NAME=13L.ISABEL&PROD=t rack_vis&PHOT=yes&ARCHIVE=active&NAV=tc&DISPLAY=Latest&STYLE=tables HTTP/1.1" 200 34907 TCP_REFRESH_MISS:DIRECT 206.48.59.87 - - [11/Sep/2003:03:32:15 -0700] "GET http://www.nrlmry.navy.mil:7777/java_scripts/popUp2.js HTTP/1.1" 304 231 TCP_IMS_HIT:NONE 67.34.38.233 - - [11/Sep/2003:03:32:16 -0700] "GET http://www.nrlmry.navy.mil:7777/sat-bin/display10.cgi?PHOT=yes&AREA=indi an/winds/scatt&PROD=qscatt&NAV=scat t_winds&CGI=scatt_winds.cgi&DISPLAY=Latest&MOSAIC_SCALE=15&CURRENT=LATES T.jpg HTTP/1.1" 200 12756 TCP_MISS:DIRECT 61.214.125.250 - - [11/Sep/2003:03:32:16 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_10_ove.jpg HTTP/1.1" 200 12179 TCP_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:16 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_09_ove.jpg HTTP/1.1" 200 12177 TCP_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:16 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_08_ove.jpg HTTP/1.1" 200 12177 TCP_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:16 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_07_ove.jpg HTTP/1.1" 200 12179 TCP_HIT:NONE 131.188.129.233 - - [11/Sep/2003:03:32:17 -0700] "GET http://www.nrlmry.navy.mil:7777/tc-bin/tc_home2.cgi?YEAR=2003&MO=Sep&BAS IN=WPAC&STORM_NAME=98W.INVEST&PR OD=track_vis&PHOT=yes&ARCHIVE=active&NAV=tc&DISPLAY=Latest&ACTIVES=03-AT L-13L.ISABEL,03-ATL-14L.NONAME,03-WPAC-15W.MAEMI,03-WPAC-98W.INVEST,03-W PAC-99W.INVEST ,&STYLE=tables HTTP/1.1" 200 30157 TCP_REFRESH_MISS:DIRECT 63.240.211.155 - - [11/Sep/2003:03:32:17 -0700] "GET http://www.nrlmry.navy.mil:7777/tc-bin/tc_display2.cgi?YEAR=2002&MO=6&BA SIN=WPAC&STORM_NAME=11W.NAKRI&PRO D=surfaceRain&PHOT=yes&DISPLAY=Prev&ARCHIVE=active&TYPE=tmi&NAV=tc&CURRE NT=20020710.0424.gms-5.vis2.x.11WNAKRI.35kts-997mb-254N-1214E.jpg&SIZE=t humb&DIR=/data /www/tropical_cyclones/tc02/WPAC/11W.NAKRI/ssmi/vis1km&MOSAIC_SCALE=25&S TYLE=frames&ACTIVES=02-ATL-08L.GUSTAV,02-WPAC-23W.HAGUPIT,02-WPAC-93W.IN VEST, HTTP/1.0 " 200 42994 TCP_MISS:DIRECT 206.48.59.87 - - [11/Sep/2003:03:32:17 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/tc_thumbs/sm200309 11.0930.meteo-7.ir_bw.x.13LISABEL.1 25kts-936mb-212N-532W.jpg HTTP/1.1" 200 9873 TCP_HIT:NONE 67.34.38.233 - - [11/Sep/2003:03:32:17 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/sat_thumbs/indian/ winds/scatt/qscatt/100_20030911.170 0.qscatt.scatt.wind.io.x.jpg HTTP/1.1" 200 141932 TCP_MISS:DIRECT 131.188.129.233 - - [11/Sep/2003:03:32:18 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/tc_thumbs/sm200309 11.0702.goes-9.vis2.x.98WINVEST. 20kts-NAmb-284N-1433E.jpg HTTP/1.1" 200 12918 TCP_HIT:NONE 63.12.141.241 - - [11/Sep/2003:03:32:22 -0700] "GET http://www.nrlmry.navy.mil:7777/java_scripts/menu.js HTTP/1.1" 304 231 TCP_IMS_HIT:NONE 63.12.141.241 - - [11/Sep/2003:03:32:23 -0700] "GET http://www.nrlmry.navy.mil:7777/java_scripts/jsClockGMT.js HTTP/1.1" 304 231 TCP_IMS_HIT:NONE 63.12.141.241 - - [11/Sep/2003:03:32:24 -0700] "GET http://www.nrlmry.navy.mil:7777/tc_pages/tc_home.html HTTP/1.1" 200 34656 TCP_IMS_HIT:NONE 63.12.141.241 - - [11/Sep/2003:03:32:24 -0700] "GET http://www.nrlmry.navy.mil:7777/java_scripts/popUp2.js HTTP/1.1" 304 230 TCP_IMS_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:25 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_10_ove.jpg HTTP/1.1" 200 20371 TCP_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:25 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_09_ove.jpg HTTP/1.1" 200 20369 TCP_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:25 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_08_ove.jpg HTTP/1.1" 200 20369 TCP_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:25 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_07_ove.jpg HTTP/1.1" 200 24469 TCP_HIT:NONE 205.213.111.51 - - [11/Sep/2003:03:32:25 -0700] "GET http://www.nrlmry.navy.mil:7777/archdat/tropical_cyclones/tc03/ATL/13L.I SABEL/ssmi/vis1km/LATEST.jpg HTTP /1.0" 200 162322 TCP_MISS:DIRECT 63.12.141.241 - - [11/Sep/2003:03:32:26 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/tc_thumbs/sm200309 11.0930.meteo-7.ir_bw.x.13LISABEL. 125kts-936mb-212N-532W.jpg HTTP/1.1" 200 9873 TCP_HIT:NONE 131.188.129.233 - - [11/Sep/2003:03:32:28 -0700] "GET http://www.nrlmry.navy.mil:7777/tc-bin/tc_home2.cgi?YEAR=2003&MO=Sep&BAS IN=WPAC&STORM_NAME=99W.INVEST&PR OD=track_vis&PHOT=yes&ARCHIVE=active&NAV=tc&DISPLAY=Latest&ACTIVES=03-AT L-13L.ISABEL,03-ATL-14L.NONAME,03-WPAC-15W.MAEMI,03-WPAC-98W.INVEST,03-W PAC-99W.INVEST ,&STYLE=tables HTTP/1.1" 200 32518 TCP_REFRESH_MISS:DIRECT 211.11.155.8 - - [11/Sep/2003:03:32:28 -0700] "GET http://www.nrlmry.navy.mil:7777/tc_pages/03_WPAC_15W.MAEMI_tmi_surfaceRa in_thumb.html HTTP/1.0" 200 36220 T CP_REFRESH_MISS:DIRECT 131.188.129.233 - - [11/Sep/2003:03:32:28 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/tc_thumbs/sm200309 11.0702.goes-9.vis2.x.99WINVEST. 15kts-NAmb-80N-1370E.jpg HTTP/1.1" 200 13425 TCP_HIT:NONE 206.48.59.87 - - [11/Sep/2003:03:32:28 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/tc_thumbs/smal1320 03.03091106.gif HTTP/1.1" 200 14276 6 TCP_HIT:NONE 209.94.201.72 - - [11/Sep/2003:03:32:32 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/sat_thumbs/pacific /eastern/south_west_ca/lowcloud/TH UMB.jpg HTTP/1.1" 200 4179 TCP_MISS:DIRECT 209.94.201.72 - - [11/Sep/2003:03:32:33 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/sat_thumbs/pacific /eastern/dycoms1/lowcloud/THUMB.jp g HTTP/1.1" 200 7174 TCP_MISS:DIRECT 209.94.201.72 - - [11/Sep/2003:03:32:33 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/sat_thumbs/pacific /eastern/north_west_us/lowcloud/TH UMB.jpg HTTP/1.1" 200 5503 TCP_MISS:DIRECT 209.94.201.72 - - [11/Sep/2003:03:32:33 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/sat_thumbs/pacific /eastern/sf_bay_area/lowcloud/THUM B.jpg HTTP/1.1" 200 3474 TCP_MISS:DIRECT 15066,1 17% 206.48.59.87 - - [11/Sep/2003:03:32:14 -0700] "GET http://www.nrlmry.navy.mil:7777/java_scripts/jsClockGMT.js HTTP/1.1" 304 231 TCP_IMS_HIT:NONE 63.101.13.176 - - [11/Sep/2003:03:32:14 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/tc_thumbs/smal1320 03.03091106.gif HTTP/1.1" 200 14276 6 TCP_HIT:NONE 206.48.59.87 - - [11/Sep/2003:03:32:15 -0700] "GET http://www.nrlmry.navy.mil:7777/tc-bin/tc_home2.cgi?YEAR=2003&MO=Sep&BAS IN=ATL&STORM_NAME=13L.ISABEL&PROD=tr ack_vis&PHOT=yes&ARCHIVE=active&NAV=tc&DISPLAY=Latest&STYLE=tables HTTP/1.1" 200 34907 TCP_REFRESH_MISS:DIRECT 206.48.59.87 - - [11/Sep/2003:03:32:15 -0700] "GET http://www.nrlmry.navy.mil:7777/java_scripts/popUp2.js HTTP/1.1" 304 231 TCP_IMS_HIT:NONE 67.34.38.233 - - [11/Sep/2003:03:32:16 -0700] "GET http://www.nrlmry.navy.mil:7777/sat-bin/display10.cgi?PHOT=yes&AREA=indi an/winds/scatt&PROD=qscatt&NAV=scatt _winds&CGI=scatt_winds.cgi&DISPLAY=Latest&MOSAIC_SCALE=15&CURRENT=LATEST .jpg HTTP/1.1" 200 12756 TCP_MISS:DIRECT 61.214.125.250 - - [11/Sep/2003:03:32:16 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_10_ove.jpg HTTP/1.1" 200 12179 TCP_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:16 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_09_ove.jpg HTTP/1.1" 200 12177 TCP_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:16 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_08_ove.jpg HTTP/1.1" 200 12177 TCP_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:16 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_07_ove.jpg HTTP/1.1" 200 12179 TCP_HIT:NONE 131.188.129.233 - - [11/Sep/2003:03:32:17 -0700] "GET http://www.nrlmry.navy.mil:7777/tc-bin/tc_home2.cgi?YEAR=2003&MO=Sep&BAS IN=WPAC&STORM_NAME=98W.INVEST&PRO D=track_vis&PHOT=yes&ARCHIVE=active&NAV=tc&DISPLAY=Latest&ACTIVES=03-ATL -13L.ISABEL,03-ATL-14L.NONAME,03-WPAC-15W.MAEMI,03-WPAC-98W.INVEST,03-WP AC-99W.INVEST,& STYLE=tables HTTP/1.1" 200 30157 TCP_REFRESH_MISS:DIRECT 63.240.211.155 - - [11/Sep/2003:03:32:17 -0700] "GET http://www.nrlmry.navy.mil:7777/tc-bin/tc_display2.cgi?YEAR=2002&MO=6&BA SIN=WPAC&STORM_NAME=11W.NAKRI&PROD =surfaceRain&PHOT=yes&DISPLAY=Prev&ARCHIVE=active&TYPE=tmi&NAV=tc&CURREN T=20020710.0424.gms-5.vis2.x.11WNAKRI.35kts-997mb-254N-1214E.jpg&SIZE=th umb&DIR=/data/w ww/tropical_cyclones/tc02/WPAC/11W.NAKRI/ssmi/vis1km&MOSAIC_SCALE=25&STY LE=frames&ACTIVES=02-ATL-08L.GUSTAV,02-WPAC-23W.HAGUPIT,02-WPAC-93W.INVE ST, HTTP/1.0" 2 00 42994 TCP_MISS:DIRECT 206.48.59.87 - - [11/Sep/2003:03:32:17 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/tc_thumbs/sm200309 11.0930.meteo-7.ir_bw.x.13LISABEL.12 5kts-936mb-212N-532W.jpg HTTP/1.1" 200 9873 TCP_HIT:NONE 67.34.38.233 - - [11/Sep/2003:03:32:17 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/sat_thumbs/indian/ winds/scatt/qscatt/100_20030911.1700 .qscatt.scatt.wind.io.x.jpg HTTP/1.1" 200 141932 TCP_MISS:DIRECT 131.188.129.233 - - [11/Sep/2003:03:32:18 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/tc_thumbs/sm200309 11.0702.goes-9.vis2.x.98WINVEST.2 0kts-NAmb-284N-1433E.jpg HTTP/1.1" 200 12918 TCP_HIT:NONE 63.12.141.241 - - [11/Sep/2003:03:32:22 -0700] "GET http://www.nrlmry.navy.mil:7777/java_scripts/menu.js HTTP/1.1" 304 231 TCP_IMS_HIT:NONE 63.12.141.241 - - [11/Sep/2003:03:32:23 -0700] "GET http://www.nrlmry.navy.mil:7777/java_scripts/jsClockGMT.js HTTP/1.1" 304 231 TCP_IMS_HIT:NONE 63.12.141.241 - - [11/Sep/2003:03:32:24 -0700] "GET http://www.nrlmry.navy.mil:7777/tc_pages/tc_home.html HTTP/1.1" 200 34656 TCP_IMS_HIT:NONE 63.12.141.241 - - [11/Sep/2003:03:32:24 -0700] "GET http://www.nrlmry.navy.mil:7777/java_scripts/popUp2.js HTTP/1.1" 304 230 TCP_IMS_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:25 -0700] "GET http://www.nrlmry.navy.mil:7777/images/proddemo_03-proddemo_10_ove.jpg HTTP/1.1" 200 20371 TCP_HIT:NONE 61.214.125.250 - - [11/Sep/2003:03:32:25 -0700] "GET http://w01.72 - - [11/Sep/2003:03:32:33 -0700] "GET http://www.nrlmry.navy.mil:7777/htdocs_dyn/thumbnails/sat_thumbs/pacific /eastern/dycoms1/lowcloud/THUMB.jpg -----Original Message----- From: Yuri N. Fominov [mailto:[EMAIL PROTECTED] Sent: Thursday, September 11, 2003 9:07 AM To: Squid-users Subject: [squid-users] Band New Squid Log Analyzer! Importance: High Dear All, I have developed SCALAR (Squid Cache Advanced Log Analyzer & Reporter). It has many of very useful reports, which definitely will help you to understand your statistics and tune Squid up for maximum cachability and performance. SCALAR is written under AWK. You can run it 'awk -f scalar.awk access.log' or make scalar.awk executable and run 'scalar.awk access.log'. By editing scalar.awk script you will be able to configure its reporting and other options. You can download SCALAR from http://www.risk.az/scalar/scalar091/scalar.awk. Try it. You feedbacks & suggestions will be highly appreciated. Regards, Yuri Fominov
