Hello David, Well, my opinion is that we should avoid redo the wheel when possible.
What I mean is that encoding translations is supported by the PostgreSQL engine on the fly in a transparent manner between tens of encoding schemes. So the simplest and fastest thing would be to simply have OMPGSQL module accept an additional parameter (client_encoding), and each time OMPGSQL open a connection to the database, the first SQL command it should send to the SQL backend should be "SET client_encoding='value'". Why re-implementing a translation module doing the same directly in Rsyslog, as it is handled by PostgreSQL? I don't see the interest except... ...if the client sending logs to the rsyslog server is so crappy that it does mix several encoding types in the same log line or even in the same block of log lines. Here, yes, you would need something "more granular" like a property replacer module... Maybe... KR. Nicolas - United Nations International Computing Center Geneva That would solve 90% of users issues related to encoding transformations. David Lang <[email protected]> a écrit : > I haven't seen any changes mentioned that would affect this, so I > doubt if anything has changed in newer versions. > > If you know that your data is in LATIN1, then I would suggest that > you look at getting a plugin written that converts all log data from > LATIN1 to UTF8. This could either be a parser module or a message > manipulation module. This should be a fairly simple module to write. > > David Lang > > On Wed, 13 Mar 2013, Nicolas HAHN wrote: > >> Hello there, >> >> I suppose plenty of you already discussed the problem I describe >> below because I've retrieved discussions speaking about that in Jan. >> 2010. >> >> In fact, as Rsyslog doesn't handle any encoding conversions (from >> SQL_ASCII to UTF8 or whatever) when configured to send logs in a >> PostgreSQL database, and it all depends of the original sender of >> the logs, that can generate a lot of errors like that in PostgreSQL: >> >> ERROR: invalid byte sequence for encoding "UTF8": 0xa0 >> >> The side effect is that in this case Rsyslog stop to send logs to >> the DB and start to spool in a directory (if configured to do so). >> >> In the various threads I've seen speaking about this problem, I >> found that at this time Rainer proposed to implement something in >> Rsyslog to handle such specific case in a way or other. Like a >> property replacer or whatever. >> >> In parallel, there were also several suggestions like running >> postgresql with SQL-ASCII encoded databases for example. Or find a >> way in the OMPGSQL module, when the connection is opened to the DB, >> to set the client_encoding variable which is a simple SQL command, >> for the time of the session. >> >> I don't know if some of you found a way to fix this kind of issue, >> if it's just impossible to have the DB in SQL-ASCII like us. >> Furthermore SQL-ASCII is clearly deprecated for UTF-8 instead. >> >> The first question: >> is the latest version of Rsyslog able to deal with encoding translations? >> >> The workaround I use personally: >> for me it's not possible to set the DB as SQL-ASCII. I've to keep it >> as UTF-8. >> Then I had to found a way for rsyslog to implement the translation >> from an encoding to another directly. >> Here is what I did: I've simply added the SET >> client_encoding='SOMETHING' in the SQL template, just before the >> INSERT command. Here is what I have by the example: >> Originally, the SQL template was like that in my config: >> >> $template mail_pgsql,"INSERT INTO es_systemevents(Message, Facility, >> FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, >> SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', >> %syslogpriority%, '%timereported:::date-rfc3339%', >> '%timegenerated:::date-rfc3339%', %iut%, '%syslogtag%')", STDSQL >> >> Now, the workaround is like that: >> >> $template es2_mail_pgsql,"set client_encoding='LATIN1';INSERT INTO >> es_systemevents(Message, Facility, FromHost, Priority, >> DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values >> ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, >> '%timereported:::date-rfc3339%', '%timegenerated:::date-rfc3339%', >> %iut%, '%syslogtag%')", STDSQL >> >> This is the way I fixed the conversion issue while keeping the >> database as UTF8. I used LATIN1 because I know my incoming data are >> in this format. You can use the same but be careful to use the >> correct encoding model for your installation. >> >> Of course, that would be really much better to have an option >> available directly at the level of the ompgsql module to set the >> client_encoding option. Something like: >> output(type="omrelp" client_encoding="LATIN1") >> in the /etc/rsyslog.conf file (that's a suggestion :-)) >> >> And now my second question: if some of you have fixed this issue >> another way (except by having a DB directly with SQL-ASCII >> encoding), could you please share how you did it? >> >> Thanks a lot for your answers. >> >> KR. >> Nicolas >> - >> United Nations International Computing Center >> Geneva >> >> >> >> ---------------------------------------------------------------- >> This message was sent using IMP, the Internet Messaging Program. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> POST if you DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST if you DON'T LIKE THAT. > ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
