I haven't seen any changes mentioned that would affect this, so I doubt if
anything has changed in newer versions.
If you know that your data is in LATIN1, then I would suggest that you look at
getting a plugin written that converts all log data from LATIN1 to UTF8. This
could either be a parser module or a message manipulation module. This should be
a fairly simple module to write.
David Lang
On Wed, 13 Mar 2013, Nicolas HAHN wrote:
Hello there,
I suppose plenty of you already discussed the problem I describe below because
I've retrieved discussions speaking about that in Jan. 2010.
In fact, as Rsyslog doesn't handle any encoding conversions (from SQL_ASCII to
UTF8 or whatever) when configured to send logs in a PostgreSQL database, and it
all depends of the original sender of the logs, that can generate a lot of
errors like that in PostgreSQL:
ERROR: invalid byte sequence for encoding "UTF8": 0xa0
The side effect is that in this case Rsyslog stop to send logs to the DB and
start to spool in a directory (if configured to do so).
In the various threads I've seen speaking about this problem, I found that at
this time Rainer proposed to implement something in Rsyslog to handle such
specific case in a way or other. Like a property replacer or whatever.
In parallel, there were also several suggestions like running postgresql with
SQL-ASCII encoded databases for example. Or find a way in the OMPGSQL module,
when the connection is opened to the DB, to set the client_encoding variable
which is a simple SQL command, for the time of the session.
I don't know if some of you found a way to fix this kind of issue, if it's just
impossible to have the DB in SQL-ASCII like us. Furthermore SQL-ASCII is
clearly deprecated for UTF-8 instead.
The first question:
is the latest version of Rsyslog able to deal with encoding translations?
The workaround I use personally:
for me it's not possible to set the DB as SQL-ASCII. I've to keep it as UTF-8.
Then I had to found a way for rsyslog to implement the translation from an
encoding to another directly.
Here is what I did: I've simply added the SET client_encoding='SOMETHING' in
the SQL template, just before the INSERT command. Here is what I have by the
example:
Originally, the SQL template was like that in my config:
$template mail_pgsql,"INSERT INTO es_systemevents(Message, Facility, FromHost,
Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%',
%syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-rfc3339%',
'%timegenerated:::date-rfc3339%', %iut%, '%syslogtag%')", STDSQL
Now, the workaround is like that:
$template es2_mail_pgsql,"set client_encoding='LATIN1';INSERT INTO
es_systemevents(Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt,
InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%,
'%timereported:::date-rfc3339%', '%timegenerated:::date-rfc3339%', %iut%,
'%syslogtag%')", STDSQL
This is the way I fixed the conversion issue while keeping the database as
UTF8. I used LATIN1 because I know my incoming data are in this format. You can
use the same but be careful to use the correct encoding model for your
installation.
Of course, that would be really much better to have an option available
directly at the level of the ompgsql module to set the client_encoding option.
Something like:
output(type="omrelp" client_encoding="LATIN1")
in the /etc/rsyslog.conf file (that's a suggestion :-))
And now my second question: if some of you have fixed this issue another way
(except by having a DB directly with SQL-ASCII encoding), could you please
share how you did it?
Thanks a lot for your answers.
KR.
Nicolas
-
United Nations International Computing Center
Geneva
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
