Hello David and Reiner, First I would like to thank you for all the help offered, I was able to setup almost everything because of you guys.
I had some issues today, though. I found that rsyslog was removing the "logger" properly but it was adding an extra empty space not sure why so I had to cut if off (by watching how to do it on video tutorial first!) by modifying the template that David gave me, I currently have it like this, $template line,"%msg:2:1000%\n" The thing here is Im not sure if this is a reliable solution, I couldnt find if there is any setting that will tell rsyslog to simply remove the empty space or to get everything until the last letter so I configured a very long (1000) number in case rsyslog cuts some part of the text. Not sure if there is any negative impact on doing it this way, if there is any other better way, please let me know. Regarding David's reply, > I would actually suggest that you use the perl script that > you posted, and > filter for programname equal to "apache_syslog", filtering > on just > 'logger' means that you can't use logger for anything > else. I ended using logger because with logger I can specify tags when sending the logs and since I have multiple vhosts and two webserver nodes I could identify from which webserver is coming the logs from, ie: :programname, isequal, "node1.www.domain.com" /var/log/httpd/node1/www.domain.com.log;line :programname, isequal, "node1.blog.domain.com" /var/log/httpd/node1/blog.domain.com.log;line :programname, isequal, "node2.www.domain.com" /var/log/httpd/node2/www.domain.com.log;line :programname, isequal, "node2.blog.domain.com" /var/log/httpd/node2/blog.domain.com.log;line I spent like 3 hours adding those lines due the amount of vhosts I have anyway this created every log and started delivering every vhost for each node on their specific log file just fine. Same thing as above Im still not really sure if this is the most realiable way of doing this with rsyslog. I hope this is the right path for doing such a big thing. Thank you again for everything. --- On Fri, 11/27/09, [email protected] <[email protected]> wrote: > From: [email protected] <[email protected]> > Subject: Re: [rsyslog] filter logger tags from syslog > To: "rsyslog-users" <[email protected]> > Date: Friday, November 27, 2009, 2:49 AM > On Thu, 26 Nov 2009, Jose Sanchez > wrote: > > > Hello, > > > > Thanks again for the great response. > > It's actually working! rsyslog is removing the > "logger:" thing and all > > the nasty stuff from it automatically, how come? Is it > because we are > > not adding any tag in the template? Im still not > understanding how > > rsyslog removes the logger thing. > > > > Ok, Im now getting the proper output and like David > said Im now getting > > issues with filtering the apache logs with all the > rsyslog messages. > > > I've tried to use the following filter but for some > reason is not > > working and Im not 100% if this is the best solution > to use, > > > > This is how I had set it up, > > > > $template line,"%msg%\n" > > if $msg contains 'GET' then > /var/log/apache.test.log;line > > *.* /var/log/test.log;line > > > > Not sure if Im on the right path, any help will be > appreciated. > > I have also tried the "if" sentence without specifying > the template name. > > > when rsyslog receives a message it parses it. the message > over the wire > hasn't changed (still has the timestamp, servername, > logger: etc), but > rsyslog puts those parts into the seperate variables and > puts what is left > of the message into the %msg% variable. > > so when you change the template from the default of > %timestamp% %hostname% %syslogtag%%msg% > to just > %msg% > > the log file has just the part you care about. > > now for the filtering. > > you could do > :%programname, isequal, "logger" > /var/log/apache.test.log;line > > (as I understand it, this format is a bit more efficiant > for rsyslog than > the equivalent of > if $programname eq "logger" then > /var/log/apache.test.log;line > ) > > I would actually suggest that you use the perl script that > you posted, and > filter for programname equal to "apache_syslog", filtering > on just > 'logger' means that you can't use logger for anything > else. > > you don't want to just filter for 'GET' as there are a > bunch of log files > that won't have GET in them > > David Lang > > > > Many Thanks. > > > > --- On Thu, 11/26/09, [email protected] <[email protected]> > wrote: > > > >> From: [email protected] <[email protected]> > >> Subject: Re: [rsyslog] filter logger tags from > syslog > >> To: "rsyslog-users" <[email protected]> > >> Date: Thursday, November 26, 2009, 6:38 PM > >> On Thu, 26 Nov 2009, Jose Sanchez > >> wrote: > >> > >>> Hello, > >>> > >>> I appreciate all the responses. > >>> Im not sure how can I can acconplish options > 1) and 2) > >> automatically. > >>> For option 3) the thing is I need "combined" > log type > >> so I cannot reform this. > >>> > >>> Im trying to centralize an access_log file > from one > >> server to the rsyslog server and I need to > completely remove > >> the tags I mentioned on my previous post. > >>> I have also tried using a perl script > mentioned at the > >> botton of this email, but it salso arriving with a > tag, > >> "apache_syslog:" as showed below, > >>> > >>> "apache_syslog: XXX.XXX.XXX.XXX - - > >> [26/Nov/2009:18:23:02 -0600] \"GET /.." > >>> > >>> Basically, this log will be parsed by awstats > which is > >> pretty much stricted with the log format so that's > why I > >> need a clean log sent from the apache server to > the rsyslog > >> server. > >> > >> don't forget that you need to filter these > messages into a > >> seperate file, > >> otherwise you will have your apache combined log > messages > >> mixed with other > >> syslog messages (which will really confuse > awstats) > >> > >> option 1 is what Rainer suggested > >> > >> option 2 is to run the log through another step > before > >> awstats runs, > >> something along the lines of > >> > >> cut -c 16- file |cut -f 3- -d ' ' |awstats > >> > >> the first cut removes the timestamp (always 15 > characters, > >> but with a > >> variable number of spaces in it), the second cut > removes > >> the servername > >> and the syslog tag ('logger:' in your first > example) > >> > >> David Lang > >> > >>> Thank you very much for all the help. > >>> > >>> Below is the Perl script: > >>> > >>> #!/usr/local/bin/perl > >>> # script: apache-access-logger > >>> > >>> use Sys::Syslog; > >>> $SERVER_NAME = shift || ''; > >>> > >>> $PRIORITY = 'info'; > >>> $FACILITY = 'local1'; > >>> > >>> Sys::Syslog::setlogsock('unix'); > >>> openlog ($SERVER_NAME,'ndelay', $FACILITY); > >>> > >>> while (<>) { > >>> chomp; > >>> syslog($PRIORITY,$_); > >>> } > >>> closelog; > >>> > >>> --- On Thu, 11/26/09, [email protected] <[email protected]> > >> wrote: > >>> > >>>> From: [email protected] <[email protected]> > >>>> Subject: Re: [rsyslog] filter logger tags > from > >> syslog > >>>> To: "rsyslog-users" <[email protected]> > >>>> Date: Thursday, November 26, 2009, 2:21 > AM > >>>> On Wed, 25 Nov 2009, Jose Sanchez > >>>> wrote: > >>>> > >>>>> Hello, > >>>>> > >>>>> I've been using classic syslog for > >> centralizing apache > >>>> access logs from one server to a remote > syslog > >> server, the > >>>> thing is syslog adds some nasty tags > before the > >> lines in the > >>>> access logs and I cant get them off, ie: > >>>>> > >>>>> "Nov 25 21:25:37 server1 logger:" > >>>>> > >>>>> I would like to know if rsyslog has > the option > >> to > >>>> filter this kind of stuff, I just want to > have the > >> logs sent > >>>> to the syslog server exactly like I was > saving > >> them in a > >>>> local access.log file. > >>>>> > >>>>> Thanks in advance. > >>>> > >>>> 'logger:' is added by the logger program > that > >> apache is > >>>> using to send the > >>>> logs to syslog. > >>>> > >>>> a properly formatted syslog message will > include > >> a > >>>> timestamp and what > >>>> server it came from (note that the apache > logs do > >> _not_ > >>>> tell you what > >>>> virtual server the log comes from, it > usually uses > >> a > >>>> different file for > >>>> each log, so when you mix them into syslog > you > >> won't be > >>>> able to tell them > >>>> apart) > >>>> > >>>> so you have three basic options > >>>> > >>>> 1. let logger do it's default thing and > then use > >> a > >>>> formatting command to > >>>> strip off the 'syslogie' parts to get back > to the > >> apache > >>>> default in the > >>>> file > >>>> > >>>> 2. leave the 'syslogie' parts in when you > write it > >> to a > >>>> file and have your > >>>> analysis tool strip them out > >>>> > >>>> 3. reformat the apache log message so that > you put > >> useful > >>>> information in > >>>> the 'syslogie' parts of the message. > >>>> > >>>> you can move the timestamp to the > beginning (you > >> can do > >>>> this with or > >>>> without the timezone, the format > obviously > >> differs > >>>> slightly) > >>>> > >>>> you can put the name of the virtual host > in the > >> server > >>>> field > >>>> > >>>> you can replace 'logger:' with something > like > >> apache[80]: > >>>> or apache[443]: > >>>> > >>>> I am going to be setting up something > along the > >> lines of #3 > >>>> in the next > >>>> few weeks. I figure I will also want to > tinker > >> with other > >>>> things in the > >>>> log message. there are items that apache > can log, > >> but does > >>>> not log by > >>>> default (I believe that how long it took > to > >> process the > >>>> request is one of > >>>> these), and also since syslog defaults to > limiting > >> log > >>>> messages to 1-2K > >>>> (depending on your impementation), there > are some > >> fields > >>>> that I would want > >>>> to move late in the message so that if > they get > >> very long > >>>> they don't cause > >>>> other fields to be lost due to truncation > (URL and > >> referrer > >>>> fields can be > >>>> several K long by themselves) > >>>> > >>>> David Lang > >>>> > _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com > >>>> > >>> > _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > -----Inline Attachment Follows----- > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
