On 09/20/2013 12:17 PM, Rainer Gerhards wrote:
On Fri, Sep 20, 2013 at 8:12 AM, Rainer Gerhards
<[email protected]>wrote:
On Fri, Sep 20, 2013 at 3:26 AM, Dave Caplinger <
[email protected]> wrote:
On Sep 18, 2013, at 2:13 PM, Axel Rau <[email protected]> wrote:
On Wed, 18 Sep 2013, David Lang wrote:
I would suggest enhancing the control character escape handling to
have a new option
EscapeInfvalidUTF8
any byte sequences that are not valid UTF8 get changed to #nnn just
like control characters.
This would help me a lot.
This will allow people to paper over many of the problems and deal
with senders that submit invalid UTF8 in the future.
After this we can talk about conversion routines.
I had a similar situation a few months back trying to get json-formated
rsyslog output (of Windows Event Logs forwarded from Snare) to be accepted
by Logstash, and this solution (enhancing the control-character-escape
handling) would have been perfectly fine with me too. No need to build the
perfect solution to tag character sets to inputs when the log sources are
going to lie anyway.
FYI: I am currently looking into a basic module that does this simple type
of "fixing". It should be fairly simple, so I guess I can implement it
today.
I have added a very small module that permits to replace control characters
with some other sequence of your liking. The patch is here:
http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=8929d2f71b76a1c3180e94f718d69ab8be01d9d6
Doc here:
http://www.rsyslog.com/doc/mmutf8fix.html
Obviously, there are many ways to improve that module. But I thought I at
least get it started and gather some feedback. If time permits, I'll add
some more functionality later today. But the basic need should be solved
(if I understood correctly ;)).
Rainer
Rainer,
thank you for providing such a quick solution to the problem!
I've just fetched the new module from git and it works nicely against
malformed test messages. Yesterday I wrote a clumsy workaround with
property replacer regular expressions, but it was quite ugly, addressed
the problem only partially and consumed substantially more cpu time.
Having a separate module is so much better solution -- I am really
grateful that you found time from your busy schedule for this.
kind regards,
risto
Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
