On Thu, 12 Feb 2015, James Lay wrote:
Hey all!
So....I've pretty much made the conversion from syslog-ng to rsyslog. I have
about 2 issues left to resolve. First, funky entries...as seen in a
different thread I am seeing some log entries that include things like:
#011
#012
#015
These are usually in between words such as the below:
#0111#011Security#01114135#011Wed Feb 11 23:08:32 2015#011540#011Security#011
The hex that shows up in the capture is 09 which is a tab character.
by default rsyslog escapes control characters see
http://www.rsyslog.com/doc/rsconf1_escape8bitcharsonreceive.html and
http://www.rsyslog.com/doc/rsconf1_escapecontrolcharactersonreceive.html Also
see the *-cc options at http://www.rsyslog.com/doc/property_replacer.html
The problem is that control characters in logs can do unexpected things to the
admin when viewing logs. The 'wrong' escape sequences can even execute arbitrary
commands in your console when you cat/grep the log file. To protect against such
things, rsyslog turns non-printable characters into #xxx where xxx is the octal
value of the character (tab -> #011)
Tab is common enough and mild enough that we really do need to create an option
that allows tab to not be modified, but it hasn't been a high enough priority
for anyone to get around to creating the patch yet.
right now, the work-around is that when rsyslog is delivering the logs to a
destination, run them through a filter to un-escape the characters, or disable
to escaping (as documented in the pages above) and accept the risk of odd stuff
happening.
Second,I have some devices which when going to rsyslog, show up as a name
like below:
<14>Feb 12 07:54:40 device-name message
all other devices show their IP as the source instead of the name, which
looks like the below:
<14>5269: Feb 12 15:54:34.474: programname: message
It looks like the message that is being sent to rsyslog is not complying with
the RFCs. Rsyslog includes a lot of heuristics to handle the most common
malformed messages, but it looks like you have some that it's not handling.
If you write out the logs with the format RSYSLOG_DebugFormat, you will see the
raw text that arrived at rsyslog, and all the variables that have been parsed
out of it.
/var/log/debuglog;RSYSLOG_DebugFormat
will do this for all logs, add filters as needed to narrow it down to the log
you are interested in.
A proper log traditional format syslog message is
<pri>timestamp hostid syslogtagmessage
(note no space between the syslogtag and the message)
pri is a 8 bit decimal number that encodes the priority and severity
timestamp is MMM DD HH:MM:SS
hostid (aka hostname) is supposed to be either the short hostname or IP address.
If rsyslog receives a message that is missing the hostid (and can detect that
it's missing because the thing in that part of the message is not a legal
hostid) it puts the IP address that it received the message from in the hostname
field
syslogtag is the program name that generated the log message, frequently with
[pid] appended to it, followed by :
message is everything else that appears in the message
In your example,
<14>5269: Feb 12 15:54:34.474: programname: message
there are two things wrong with this log
1. the seconds has fractional seconds added to it
2. it has a number: after the pri section before the timestamp.
Now, when you run into this sort of problem, what can you do.
Ideally you go to the vendor that's doing this and tell them they are violating
the RFCs, and get them to fix it (tell you how to reconfigure the system to send
legal messages, fix it in an update, etc)
If you can't do this (or can't wait for the patch), what you can do is detect
the problem in rsyslog and then use the template engine to craft a replacement
log message when you deliver the message.
This is one reason I _really_ like to deploy a two layer syslog infrastructure.
The first layer receives logs from everything and does this sort of cleanup
(which can be fairly nasty and cpu intensive) and then the second layer is
guaranteed to be receiving 'clean' messages and can concentrate on
processing/delivering them.
Another option is to craft the replacement message and deliver it to yourself
via localhost and then throw away the original message.
I could go on longer, but I'll let you ask for more info as you need it.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
