On 2015-02-12 12:18 PM, David Lang wrote:
On Thu, 12 Feb 2015, James Lay wrote:
Hey all!
So....I've pretty much made the conversion from syslog-ng to
rsyslog. I have about 2 issues left to resolve. First, funky
entries...as seen in a different thread I am seeing some log entries
that include things like:
#011
#012
#015
These are usually in between words such as the below:
#0111#011Security#01114135#011Wed Feb 11 23:08:32
2015#011540#011Security#011
The hex that shows up in the capture is 09 which is a tab character.
by default rsyslog escapes control characters see
http://www.rsyslog.com/doc/rsconf1_escape8bitcharsonreceive.html and
http://www.rsyslog.com/doc/rsconf1_escapecontrolcharactersonreceive.html
Also see the *-cc options at
http://www.rsyslog.com/doc/property_replacer.html
The problem is that control characters in logs can do unexpected
things to the admin when viewing logs. The 'wrong' escape sequences
can even execute arbitrary commands in your console when you cat/grep
the log file. To protect against such things, rsyslog turns
non-printable characters into #xxx where xxx is the octal value of
the
character (tab -> #011)
Tab is common enough and mild enough that we really do need to create
an option that allows tab to not be modified, but it hasn't been a
high enough priority for anyone to get around to creating the patch
yet.
right now, the work-around is that when rsyslog is delivering the
logs to a destination, run them through a filter to un-escape the
characters, or disable to escaping (as documented in the pages above)
and accept the risk of odd stuff happening.
Second,I have some devices which when going to rsyslog, show up as a
name like below:
<14>Feb 12 07:54:40 device-name message
all other devices show their IP as the source instead of the name,
which looks like the below:
<14>5269: Feb 12 15:54:34.474: programname: message
It looks like the message that is being sent to rsyslog is not
complying with the RFCs. Rsyslog includes a lot of heuristics to
handle the most common malformed messages, but it looks like you have
some that it's not handling.
If you write out the logs with the format RSYSLOG_DebugFormat, you
will see the raw text that arrived at rsyslog, and all the variables
that have been parsed out of it.
/var/log/debuglog;RSYSLOG_DebugFormat
will do this for all logs, add filters as needed to narrow it down to
the log you are interested in.
A proper log traditional format syslog message is
<pri>timestamp hostid syslogtagmessage
(note no space between the syslogtag and the message)
pri is a 8 bit decimal number that encodes the priority and severity
timestamp is MMM DD HH:MM:SS
hostid (aka hostname) is supposed to be either the short hostname or
IP address. If rsyslog receives a message that is missing the hostid
(and can detect that it's missing because the thing in that part of
the message is not a legal hostid) it puts the IP address that it
received the message from in the hostname field
syslogtag is the program name that generated the log message,
frequently with [pid] appended to it, followed by :
message is everything else that appears in the message
In your example,
<14>5269: Feb 12 15:54:34.474: programname: message
there are two things wrong with this log
1. the seconds has fractional seconds added to it
2. it has a number: after the pri section before the timestamp.
Now, when you run into this sort of problem, what can you do.
Ideally you go to the vendor that's doing this and tell them they are
violating the RFCs, and get them to fix it (tell you how to
reconfigure the system to send legal messages, fix it in an update,
etc)
If you can't do this (or can't wait for the patch), what you can do
is detect the problem in rsyslog and then use the template engine to
craft a replacement log message when you deliver the message.
This is one reason I _really_ like to deploy a two layer syslog
infrastructure. The first layer receives logs from everything and
does
this sort of cleanup (which can be fairly nasty and cpu intensive)
and
then the second layer is guaranteed to be receiving 'clean' messages
and can concentrate on processing/delivering them.
Another option is to craft the replacement message and deliver it to
yourself via localhost and then throw away the original message.
I could go on longer, but I'll let you ask for more info as you need
it.
David Lang
Wow thanks SO much David...really...that helps. I think for #2 I'm
just going to mod my scripts to remove the cruft and drive on..easy. As
for the devices that aren't syslog RFC compliant, I think I'll just
leave them be and work around it in my report scripts as well. Thanks
again so much...really appreciate it.
James
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
