On Thu, 26 Nov 2009, Jose Sanchez wrote:
Hello,
Thanks again for the great response.
It's actually working! rsyslog is removing the "logger:" thing and all
the nasty stuff from it automatically, how come? Is it because we are
not adding any tag in the template? Im still not understanding how
rsyslog removes the logger thing.
Ok, Im now getting the proper output and like David said Im now getting
issues with filtering the apache logs with all the rsyslog messages.
I've tried to use the following filter but for some reason is not
working and Im not 100% if this is the best solution to use,
This is how I had set it up,
$template line,"%msg%\n"
if $msg contains 'GET' then /var/log/apache.test.log;line
*.* /var/log/test.log;line
Not sure if Im on the right path, any help will be appreciated.
I have also tried the "if" sentence without specifying the template name.
when rsyslog receives a message it parses it. the message over the wire
hasn't changed (still has the timestamp, servername, logger: etc), but
rsyslog puts those parts into the seperate variables and puts what is left
of the message into the %msg% variable.
so when you change the template from the default of
%timestamp% %hostname% %syslogtag%%msg%
to just
%msg%
the log file has just the part you care about.
now for the filtering.
you could do
:%programname, isequal, "logger" /var/log/apache.test.log;line
(as I understand it, this format is a bit more efficiant for rsyslog than
the equivalent of
if $programname eq "logger" then /var/log/apache.test.log;line
)
I would actually suggest that you use the perl script that you posted, and
filter for programname equal to "apache_syslog", filtering on just
'logger' means that you can't use logger for anything else.
you don't want to just filter for 'GET' as there are a bunch of log files
that won't have GET in them
David Lang
Many Thanks.
--- On Thu, 11/26/09, [email protected] <[email protected]> wrote:
From: [email protected] <[email protected]>
Subject: Re: [rsyslog] filter logger tags from syslog
To: "rsyslog-users" <[email protected]>
Date: Thursday, November 26, 2009, 6:38 PM
On Thu, 26 Nov 2009, Jose Sanchez
wrote:
Hello,
I appreciate all the responses.
Im not sure how can I can acconplish options 1) and 2)
automatically.
For option 3) the thing is I need "combined" log type
so I cannot reform this.
Im trying to centralize an access_log file from one
server to the rsyslog server and I need to completely remove
the tags I mentioned on my previous post.
I have also tried using a perl script mentioned at the
botton of this email, but it salso arriving with a tag,
"apache_syslog:" as showed below,
"apache_syslog: XXX.XXX.XXX.XXX - -
[26/Nov/2009:18:23:02 -0600] \"GET /.."
Basically, this log will be parsed by awstats which is
pretty much stricted with the log format so that's why I
need a clean log sent from the apache server to the rsyslog
server.
don't forget that you need to filter these messages into a
seperate file,
otherwise you will have your apache combined log messages
mixed with other
syslog messages (which will really confuse awstats)
option 1 is what Rainer suggested
option 2 is to run the log through another step before
awstats runs,
something along the lines of
cut -c 16- file |cut -f 3- -d ' ' |awstats
the first cut removes the timestamp (always 15 characters,
but with a
variable number of spaces in it), the second cut removes
the servername
and the syslog tag ('logger:' in your first example)
David Lang
Thank you very much for all the help.
Below is the Perl script:
#!/usr/local/bin/perl
# script: apache-access-logger
use Sys::Syslog;
$SERVER_NAME = shift || '';
$PRIORITY = 'info';
$FACILITY = 'local1';
Sys::Syslog::setlogsock('unix');
openlog ($SERVER_NAME,'ndelay', $FACILITY);
while (<>) {
chomp;
syslog($PRIORITY,$_);
}
closelog;
--- On Thu, 11/26/09, [email protected] <[email protected]>
wrote:
From: [email protected] <[email protected]>
Subject: Re: [rsyslog] filter logger tags from
syslog
To: "rsyslog-users" <[email protected]>
Date: Thursday, November 26, 2009, 2:21 AM
On Wed, 25 Nov 2009, Jose Sanchez
wrote:
Hello,
I've been using classic syslog for
centralizing apache
access logs from one server to a remote syslog
server, the
thing is syslog adds some nasty tags before the
lines in the
access logs and I cant get them off, ie:
"Nov 25 21:25:37 server1 logger:"
I would like to know if rsyslog has the option
to
filter this kind of stuff, I just want to have the
logs sent
to the syslog server exactly like I was saving
them in a
local access.log file.
Thanks in advance.
'logger:' is added by the logger program that
apache is
using to send the
logs to syslog.
a properly formatted syslog message will include
a
timestamp and what
server it came from (note that the apache logs do
_not_
tell you what
virtual server the log comes from, it usually uses
a
different file for
each log, so when you mix them into syslog you
won't be
able to tell them
apart)
so you have three basic options
1. let logger do it's default thing and then use
a
formatting command to
strip off the 'syslogie' parts to get back to the
apache
default in the
file
2. leave the 'syslogie' parts in when you write it
to a
file and have your
analysis tool strip them out
3. reformat the apache log message so that you put
useful
information in
the 'syslogie' parts of the message.
you can move the timestamp to the beginning (you
can do
this with or
without the timezone, the format obviously
differs
slightly)
you can put the name of the virtual host in the
server
field
you can replace 'logger:' with something like
apache[80]:
or apache[443]:
I am going to be setting up something along the
lines of #3
in the next
few weeks. I figure I will also want to tinker
with other
things in the
log message. there are items that apache can log,
but does
not log by
default (I believe that how long it took to
process the
request is one of
these), and also since syslog defaults to limiting
log
messages to 1-2K
(depending on your impementation), there are some
fields
that I would want
to move late in the message so that if they get
very long
they don't cause
other fields to be lost due to truncation (URL and
referrer
fields can be
several K long by themselves)
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
