On 09/27/2013 02:33 PM, Risto Vaarandi wrote:
On 09/20/2013 06:29 PM, Rainer Gerhards wrote:
On Fri, Sep 20, 2013 at 12:01 PM, Axel Rau <[email protected]> wrote:
Obviously, there are many ways to improve that module. But I thought
I at
least get it started and gather some feedback. If time permits, I'll
add
some more functionality later today. But the basic need should be
solved
(if I understood correctly ;)).
I think, the "basic need" would be to replace only invalid UTF-8
sequences, not every character with code < 32 or > 126.
thanks. I have just committed to master branch a version who does this
(by
default).
Does the restriction to IPv4 apply only to expressions like
if $fromhost-ip == "10.0.0.1" then
or in general?
That was a leftover from the doc I used as basis, mmutf8fix doesn't care
about IPvx. It's removed now.
@risto: you now need to specify mode="controlcharacters" to replace all
non-printable US-ASCII characters, as proper UTF-8 checks are default
now.
I thought that's more appropriate for this module ;)
See last sample in doc:
http://www.rsyslog.com/doc/mmutf8fix.html
Feedback on this module is appreciated.
I have had it running for about a week and so far it has been able to
write all log messages into Elasticsearch without issues (many millions
of messages per day). Looks like it is working just as expected.
kind regards,
risto
Actually, yesterday there was one write failure into elasticsearch.
I had rsyslogd running with
action(type="mmutf8fix" replacementChar="_")
statement which accepts not only us-ascii, but also all utf8 characters.
The log message was badly malformed and a number of replacements were
done (my replacement character is "_" as you can see from above
statement). However, in the very end of the log message the final byte
was a non-utf8 character, and that was left unreplaced. The last 4 bytes
of the message look like follows:
<us-ascii char><us-ascii char><replaced char><unreplaced non-utf8 char>
The error message produced by elasticsearch looks like follows:
Invalid UTF-8 middle byte 0x22
The issue is not very urgent, because I mostly care about us-ascii
characters and can thus enable mode="controlcharacters" for a workaround.
kind regards,
risto
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
