On Fri, Sep 20, 2013 at 2:02 PM, Risto Vaarandi <[email protected]>wrote:
> On 09/20/2013 12:17 PM, Rainer Gerhards wrote: > >> On Fri, Sep 20, 2013 at 8:12 AM, Rainer Gerhards >> <[email protected]>**wrote: >> >> On Fri, Sep 20, 2013 at 3:26 AM, Dave Caplinger < >>> [email protected]> wrote: >>> >>> On Sep 18, 2013, at 2:13 PM, Axel Rau <[email protected]> wrote: >>>> >>>> On Wed, 18 Sep 2013, David Lang wrote: >>>>>> >>>>>> I would suggest enhancing the control character escape handling to >>>>>> >>>>> have a new option >>>> >>>>> >>>>>> EscapeInfvalidUTF8 >>>>>> >>>>>> any byte sequences that are not valid UTF8 get changed to #nnn just >>>>>> >>>>> like control characters. >>>> >>>>> This would help me a lot. >>>>> >>>>>> >>>>>> This will allow people to paper over many of the problems and deal >>>>>> >>>>> with senders that submit invalid UTF8 in the future. >>>> >>>>> >>>>>> After this we can talk about conversion routines. >>>>>> >>>>> >>>> I had a similar situation a few months back trying to get json-formated >>>> rsyslog output (of Windows Event Logs forwarded from Snare) to be >>>> accepted >>>> by Logstash, and this solution (enhancing the control-character-escape >>>> handling) would have been perfectly fine with me too. No need to build >>>> the >>>> perfect solution to tag character sets to inputs when the log sources >>>> are >>>> going to lie anyway. >>>> >>>> >>>> >>>> FYI: I am currently looking into a basic module that does this simple >>> type >>> of "fixing". It should be fairly simple, so I guess I can implement it >>> today. >>> >>> >> I have added a very small module that permits to replace control >> characters >> with some other sequence of your liking. The patch is here: >> >> http://git.adiscon.com/?p=**rsyslog.git;a=commitdiff;h=** >> 8929d2f71b76a1c3180e94f718d69a**b8be01d9d6<http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=8929d2f71b76a1c3180e94f718d69ab8be01d9d6> >> >> Doc here: >> http://www.rsyslog.com/doc/**mmutf8fix.html<http://www.rsyslog.com/doc/mmutf8fix.html> >> >> Obviously, there are many ways to improve that module. But I thought I at >> least get it started and gather some feedback. If time permits, I'll add >> some more functionality later today. But the basic need should be solved >> (if I understood correctly ;)). >> >> Rainer >> > > Rainer, > thank you for providing such a quick solution to the problem! > my pleasure, it's good that you brought the problem up. > I've just fetched the new module from git and it works nicely against > malformed test messages. Yesterday I wrote a clumsy workaround with > property replacer regular expressions, but it was quite ugly, addressed the > problem only partially and consumed substantially more cpu time. > Having a separate module is so much better solution -- I am really > grateful that you found time from your busy schedule for this. > great we have solved one problem. But I agree to Axel, it's probably needs a pure UTF-8 malformed test to be useful for many more folks. Will see if I can add at least this mode quickly. Rainer > kind regards, > risto > > >> >> >>> Rainer >>> >>> >>> ______________________________**_________________ >> rsyslog mailing list >> http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> >> http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> >> > ______________________________**_________________ > rsyslog mailing list > http://lists.adiscon.net/**mailman/listinfo/rsyslog<http://lists.adiscon.net/mailman/listinfo/rsyslog> > http://www.rsyslog.com/**professional-services/<http://www.rsyslog.com/professional-services/> > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
