Am 10.10.2013 um 14:46 schrieb Rainer Gerhards <[email protected]>:
> On Thu, Oct 3, 2013 at 9:09 AM, Risto Vaarandi <[email protected]>wrote: > >> On 09/27/2013 02:33 PM, Risto Vaarandi wrote: >> >>> On 09/20/2013 06:29 PM, Rainer Gerhards wrote: >>> >>>> On Fri, Sep 20, 2013 at 12:01 PM, Axel Rau <[email protected]> wrote: >>>> >>>> Obviously, there are many ways to improve that module. But I thought >>>>>> I at >>>>>> least get it started and gather some feedback. If time permits, I'll >>>>>> add >>>>>> some more functionality later today. But the basic need should be >>>>>> solved >>>>>> (if I understood correctly ;)). >>>>>> >>>>> I think, the "basic need" would be to replace only invalid UTF-8 >>>>> sequences, not every character with code < 32 or > 126. >>>>> >>>>> >>>> thanks. I have just committed to master branch a version who does this >>>> (by >>>> default). >>>> >>>> >>>> Does the restriction to IPv4 apply only to expressions like >>>>> if $fromhost-ip == "10.0.0.1" then >>>>> or in general? >>>>> >>>>> >>>> That was a leftover from the doc I used as basis, mmutf8fix doesn't care >>>> about IPvx. It's removed now. >>>> >>>> @risto: you now need to specify mode="controlcharacters" to replace all >>>> non-printable US-ASCII characters, as proper UTF-8 checks are default >>>> now. >>>> I thought that's more appropriate for this module ;) >>>> >>>> See last sample in doc: >>>> http://www.rsyslog.com/doc/**mmutf8fix.html<http://www.rsyslog.com/doc/mmutf8fix.html> >>>> >>>> Feedback on this module is appreciated. >>>> >>> >>> I have had it running for about a week and so far it has been able to >>> write all log messages into Elasticsearch without issues (many millions >>> of messages per day). Looks like it is working just as expected. >>> kind regards, >>> risto >>> >>> >> Actually, yesterday there was one write failure into elasticsearch. >> I had rsyslogd running with >> >> action(type="mmutf8fix" replacementChar="_") >> >> statement which accepts not only us-ascii, but also all utf8 characters. >> >> The log message was badly malformed and a number of replacements were done >> (my replacement character is "_" as you can see from above statement). >> However, in the very end of the log message the final byte was a non-utf8 >> character, and that was left unreplaced. The last 4 bytes of the message >> look like follows: >> >> <us-ascii char><us-ascii char><replaced char><unreplaced non-utf8 char> >> >> The error message produced by elasticsearch looks like follows: >> Invalid UTF-8 middle byte 0x22 >> >> The issue is not very urgent, because I mostly care about us-ascii >> characters and can thus enable mode="controlcharacters" for a workaround. >> >> > I finally fixed this border case, patch here: > > http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=97bda43e372a506671cb7007b6041e4160a02b04 > > I would appreciate if you could apply and test the patch, as I had only > time to do a very quick test (plus obviously nothing beats high-volume real > traffic ;)). > I found another border case with 2-byte-sequence, where 0xc0 and 0xc1 are not allowed: - - - diff --git a/plugins/mmutf8fix/mmutf8fix.c b/plugins/mmutf8fix/mmutf8fix.c index 351bb12..65739ed 100644 --- a/plugins/mmutf8fix/mmutf8fix.c +++ b/plugins/mmutf8fix/mmutf8fix.c @@ -254,9 +254,14 @@ doUTF8(instanceData *pData, uchar *msg, int lenMsg) ; /* nothing to do, all well */ } else if((c & 0xe0) == 0xc0) { /* 2-byte sequence */ - strtIdx = i; - seqLen = bytesLeft = 1; - codepoint = c & 0x1f; + /* 0xc0 and 0xc1 are illegal */ + if (c == 0xc0 || c == 0xc1) { + msg[i] = pData->replChar; + } else { + strtIdx = i; + seqLen = bytesLeft = 1; + codepoint = c & 0x1f; + } } else if((c & 0xf0) == 0xe0) { /* 3-byte sequence */ strtIdx = i; - - - Regards, Axel --- PGP-Key:29E99DD6 ☀ +49 151 2300 9283 ☀ computing @ chaos claudius _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
