Re: analyzing httpd-error.log
Charles Howse skrev: On Mar 28, 2009, at 11:51 PM, Olivier Nicole wrote: Hi, Webalizer is doing what it's supposed to with httpd-access.log, but when I give it the error log to process is coughs, spits and spills out errors with no data processed. My research hasn't turned up a good solution for webalizer and -error.log. The format of error log is pretty much different from the format of transfer log. No wonder webalizer is not liking it. You may have to write your own format for th error log. Well, can anyone suggest a port that will parse the error.log and output it to a web page that's easy to read? Also, in httpd.conf what level of detail should I set in the error.log to get the most information. It's currently set to 'warn', which I understand to be 'warn' and everything more critical than that. I don't care about the size of the log, or the amount of garbage per line. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org Hi Charles, I don't know if this will help you but try looking at Lire (logreport.org). For just collecting and web-based viewing phplogcon from www.phplogcon.org can be used. /R ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: analyzing httpd-error.log
The problem with Apache's error.log is that there is no standard format. Error messages generated by Apache itself are somewhat standardized, but messages from third-party modules are not. All kind of things will end up in the error.log, including stuff written to stdout by CGI programs, such as perl error messages, exception traces from Python programs (usually multiple lines each), and so on. There is no simple way to reliably parse and analyze all of that completely automatically. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing b. M. Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung: secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün- chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd If you think C++ is not overly complicated, just what is a protected abstract virtual base pure virtual private destructor, and when was the last time you needed one? -- Tom Cargil, C++ Journal ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: analyzing httpd-error.log
On Sunday 29 March 2009 22:25:55 Charles Howse wrote: On Mar 29, 2009, at 1:54 PM, Glen Barber wrote: On Sun, Mar 29, 2009 at 1:33 PM, Charles Howse cho...@charter.net wrote: On Mar 28, 2009, at 11:51 PM, Olivier Nicole wrote: Hi, Webalizer is doing what it's supposed to with httpd-access.log, but when I give it the error log to process is coughs, spits and spills out errors with no data processed. My research hasn't turned up a good solution for webalizer and -error.log. What are the errors? Intrusion attempts, (a few) bad links in my website, also I use the error.log to troubleshoot cgi scripts. Nothing beats tail -f for debugging. 404's can be gathered from access log, by webalizer (or awstats or ...) and are summarized when enabled. If I remember correctly, there's also a referer top list, that specifies which pages link to invalid pages, but it may have been awstats that does this. Can't think of anything specific for apache error log, I roll my own grok rules if I'm really interested in a specific vulnerability. See sysutils/grok. -- Mel ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: analyzing httpd-error.log
On Mar 30, 2009, at 3:11 PM, Mel Flynn wrote: On Sunday 29 March 2009 22:25:55 Charles Howse wrote: On Mar 29, 2009, at 1:54 PM, Glen Barber wrote: On Sun, Mar 29, 2009 at 1:33 PM, Charles Howse cho...@charter.net wrote: On Mar 28, 2009, at 11:51 PM, Olivier Nicole wrote: Hi, Webalizer is doing what it's supposed to with httpd-access.log, but when I give it the error log to process is coughs, spits and spills out errors with no data processed. My research hasn't turned up a good solution for webalizer and -error.log. What are the errors? Intrusion attempts, (a few) bad links in my website, also I use the error.log to troubleshoot cgi scripts. Nothing beats tail -f for debugging. 404's can be gathered from access log, by webalizer (or awstats or ...) and are summarized when enabled. If I remember correctly, there's also a referer top list, that specifies which pages link to invalid pages, but it may have been awstats that does this. Can't think of anything specific for apache error log, I roll my own grok rules if I'm really interested in a specific vulnerability. See sysutils/grok. Yes, awstats does a fair job of summarizing 404's. I have a hardware router, not extremely interested in a packet filter for my webserver. I guess I could do some deny from rules in apache.conf if necessary...? * #!/bin/sh date=`date +%b %e` cd /var/log grep $date httpd-error.log | grep -v 192.168.254.254 | grep -v 192.168.254.3 /root/err.log mail -s httpd-error.log charles /root/err.log * I run this from cron @ 11:59 PM every night, and that gives me the error log for that day. Maybe I won't miss anything serious between 11:59 and midnight. Can anyone help with a grep command that will filter out all addresses beginning with 192.168.254? Something to replace the 2 piped commands with only 1. I tried: grep -v -regexp[^192\.168\.254\.] but that didn't match any records...? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: analyzing httpd-error.log
On Monday 30 March 2009 22:55:35 Charles Howse wrote: grep $date httpd-error.log | grep -v 192.168.254.254 | grep -v 192.168.254.3 /root/err.log Using pcregrep, installed by devel/pcre, typically available on apache systems: pcregrep $date.* \[client (?!192\.168\.254) /var/log/httpd-error.log | \ mail -sError log for $date charles Should work. As far as I know, look ahead assertions aren't supported by re(7) or the extended version. -- Mel ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: analyzing httpd-error.log
On Mar 30, 2009, at 4:09 PM, Mel Flynn wrote: On Monday 30 March 2009 22:55:35 Charles Howse wrote: grep $date httpd-error.log | grep -v 192.168.254.254 | grep -v 192.168.254.3 /root/err.log Using pcregrep, installed by devel/pcre, typically available on apache systems: pcregrep $date.* \[client (?!192\.168\.254) /var/log/httpd- error.log | \ mail -sError log for $date charles Should work. As far as I know, look ahead assertions aren't supported by re(7) or the extended version. Sweet! Look at the intrusion attempt by 130.79.37.213 (it's from the Universite de Strasbourg in France): [Mon Mar 30 00:02:24 2009] [error] [client 72.14.199.109] File does not exist: /usr/local/www/apache22/data/blog.rss [Mon Mar 30 00:13:26 2009] [error] [client 130.79.37.213] script not found or unable to stat: /usr/local/www/apache22/cgi-bin/awstats [Mon Mar 30 00:13:26 2009] [error] [client 130.79.37.213] script not found or unable to stat: /usr/local/www/apache22/cgi-bin/awstats.pl [Mon Mar 30 00:13:27 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/cgi [Mon Mar 30 00:13:29 2009] [error] [client 130.79.37.213] script not found or unable to stat: /usr/local/www/apache22/cgi-bin/stats [Mon Mar 30 00:13:29 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/stats [Mon Mar 30 00:13:30 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/awstats.pl [Mon Mar 30 00:13:30 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/cgi [Mon Mar 30 00:13:30 2009] [error] [client 130.79.37.213] script not found or unable to stat: /usr/local/www/apache22/cgi-bin/awstats.pl [Mon Mar 30 00:13:31 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scgi-bin [Mon Mar 30 00:13:31 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scgi-bin [Mon Mar 30 00:13:31 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/cgi [Mon Mar 30 00:13:32 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scgi [Mon Mar 30 00:13:32 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scripts [Mon Mar 30 00:13:32 2009] [error] [client 130.79.37.213] script not found or unable to stat: /usr/local/www/apache22/cgi-bin/awstats [Mon Mar 30 00:13:33 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scgi-bin [Mon Mar 30 00:13:33 2009] [error] [client 130.79.37.213] script not found or unable to stat: /usr/local/www/apache22/cgi-bin/stats [Mon Mar 30 00:13:33 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scgi-bin [Mon Mar 30 00:13:34 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/stats [Mon Mar 30 00:13:34 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/hints.pl [Mon Mar 30 00:13:34 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/cgi [Mon Mar 30 00:13:34 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scgi [Mon Mar 30 00:13:35 2009] [error] [client 130.79.37.213] script not found or unable to stat: /usr/local/www/apache22/cgi-bin/hints.pl [Mon Mar 30 00:13:35 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scgi-bin [Mon Mar 30 00:13:35 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/hints [Mon Mar 30 00:13:36 2009] [error] [client 130.79.37.213] script not found or unable to stat: /usr/local/www/apache22/cgi-bin/hints [Mon Mar 30 00:13:36 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scgi-bin [Mon Mar 30 00:13:36 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/webhints [Mon Mar 30 00:13:37 2009] [error] [client 130.79.37.213] script not found or unable to stat: /usr/local/www/apache22/cgi-bin/webhints [Mon Mar 30 00:13:37 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scgi-bin [Mon Mar 30 00:13:37 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/hints.cgi [Mon Mar 30 00:13:38 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/cgi [Mon Mar 30 00:13:38 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scgi [Mon Mar 30 00:13:38 2009] [error] [client 130.79.37.213] script not found or unable to stat: /usr/local/www/apache22/cgi-bin/hints.cgi [Mon Mar 30 00:13:39 2009] [error] [client 130.79.37.213] File does not exist: /usr/local/www/apache22/data/scgi-bin [Mon Mar 30 00:13:39 2009] [error] [client 130.79.37.213] File does not exist:
Re: analyzing httpd-error.log
On Mar 28, 2009, at 11:51 PM, Olivier Nicole wrote: Hi, Webalizer is doing what it's supposed to with httpd-access.log, but when I give it the error log to process is coughs, spits and spills out errors with no data processed. My research hasn't turned up a good solution for webalizer and -error.log. The format of error log is pretty much different from the format of transfer log. No wonder webalizer is not liking it. You may have to write your own format for th error log. Well, can anyone suggest a port that will parse the error.log and output it to a web page that's easy to read? Also, in httpd.conf what level of detail should I set in the error.log to get the most information. It's currently set to 'warn', which I understand to be 'warn' and everything more critical than that. I don't care about the size of the log, or the amount of garbage per line. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: analyzing httpd-error.log
On Sun, Mar 29, 2009 at 1:33 PM, Charles Howse cho...@charter.net wrote: On Mar 28, 2009, at 11:51 PM, Olivier Nicole wrote: Hi, Webalizer is doing what it's supposed to with httpd-access.log, but when I give it the error log to process is coughs, spits and spills out errors with no data processed. My research hasn't turned up a good solution for webalizer and -error.log. What are the errors? The format of error log is pretty much different from the format of transfer log. No wonder webalizer is not liking it. You may have to write your own format for th error log. Well, can anyone suggest a port that will parse the error.log and output it to a web page that's easy to read? Webalizer is probably your best bet. Also, in httpd.conf what level of detail should I set in the error.log to get the most information. It's currently set to 'warn', which I understand to be 'warn' and everything more critical than that. I don't care about the size of the log, or the amount of garbage per line. The 'debug' log level will provide the most verbosity. -- Glen Barber ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: analyzing httpd-error.log
On Mar 29, 2009, at 1:54 PM, Glen Barber wrote: On Sun, Mar 29, 2009 at 1:33 PM, Charles Howse cho...@charter.net wrote: On Mar 28, 2009, at 11:51 PM, Olivier Nicole wrote: Hi, Webalizer is doing what it's supposed to with httpd-access.log, but when I give it the error log to process is coughs, spits and spills out errors with no data processed. My research hasn't turned up a good solution for webalizer and -error.log. What are the errors? Intrusion attempts, (a few) bad links in my website, also I use the error.log to troubleshoot cgi scripts. The format of error log is pretty much different from the format of transfer log. No wonder webalizer is not liking it. You may have to write your own format for th error log. Well, can anyone suggest a port that will parse the error.log and output it to a web page that's easy to read? Webalizer is probably your best bet. Also, in httpd.conf what level of detail should I set in the error.log to get the most information. It's currently set to 'warn', which I understand to be 'warn' and everything more critical than that. I don't care about the size of the log, or the amount of garbage per line. The 'debug' log level will provide the most verbosity. Thanks, Glen. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
analyzing httpd-error.log
Hi, I'm running 6.4-STABLE, Apache22, logwatch and Webalizer. Webalizer is doing what it's supposed to with httpd-access.log, but when I give it the error log to process is coughs, spits and spills out errors with no data processed. My research hasn't turned up a good solution for webalizer and -error.log. In httpd.conf, I'm using both common and combined log formats. Webalizer will read them both. With logwatch, and even with the log detail turned up to 7, I'm not getting the detail I want from the error.log. I want to see *everything* in that log. Anybody got a solution? -- Thanks, Charles Things that make you say, Hmm... If you were a pastor, and you were getting married, would you hire a pastor, or would you do the wedding yourself? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: analyzing httpd-error.log
Hi, Webalizer is doing what it's supposed to with httpd-access.log, but when I give it the error log to process is coughs, spits and spills out errors with no data processed. My research hasn't turned up a good solution for webalizer and -error.log. The format of error log is pretty much different from the format of transfer log. No wonder webalizer is not liking it. You may have to write your own format for th error log. Bests, Olivier ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
