V5 has the capability to discard messages that cause an action failure. However, this is mostly untested yet, AND the action must support it by providing proper status information - it must differentiate between system-induced errors (which can be retried) and message-induced errors (which need the discard). ompgsql currently does not provide that status information.
Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Jakob Haufe > Sent: Wednesday, January 20, 2010 7:21 PM > To: [email protected] > Subject: Re: [rsyslog] PostgreSQL: Problems with character encoding > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Wed, 6 Jan 2010 16:14:59 +0100 > Marc Schiffbauer <[email protected]> wrote: > > > which encoding should be chosen for the database when using postgres? > > As far as I understand the syslog protocol (at least the legacy one), > it has > no concept of character encodings at all. So if you simply want to > make sure > that everything ends up in the database "as is", then choose SQL_ASCII. > > > My rsyslog version is 4.4.3. > > > > Which client_encoding does rsyslog use in ompgsql? > > Right now, it does net set an encoding by itself, so the database > default > applies. If I'm not mistaken, you can even set that per user from > inside of > postgres. So I would rather vote against another configuration > parameter here. > > > I currently have set UTF-8 on the database. It worked for a while > until > > some special message arrived at the server where postgres denies the > INSERT: > > > > 2010-01-06 16:13:11 CET syslog syslog ERROR: invalid byte sequence > for > > encoding "UTF8": 0xd220 > > 2010-01-06 16:13:11 CET syslog syslog HINT: This error can also > happen if > > the byte sequence does not match the encoding expected by the server, > which > > is controlled by "client_encoding". > > Were you able to isolate the message? Or find out which program was > sending > it? > > > Now rsyslog is not able to log anything... it is currently spooling > to disk > > because it "hangs" at this message not being accepted by postgres. > > This is bad, because if the machine is an open syslog server that > simply > collects everything it gets, we have a potential DoS vector here. > > I can think of three options: > > * Drop the message and report that we did so. That would be rather > easy, > but might not be what people want. > > * Re-insert the message after converting it from ASCII to UTF-8 or > whatever > the DB encoding is. But this might/will produce garbage if the input > is not > ASCII. It also creates more load on the system if these messages are > frequent. Guessing the input encoding is hard or even impossible, > depending > on the set you guess from. > > * Make the database SQL_ASCII. This will silently accept anything but > will > create nonsense from UTF/UCS encoded messages. Also might create > trouble > for programs like phplogcon that analyze the logs. > > For me, this sums up to one question: > > Can we make ompgsql UTF/UCS-clean and at the same time not choke on > non-UTF8 > strings? Everyone is trying to be UTF-8 clean these days, so it would > be bad > if ompgsql could not keep up. > > Comments please. > > Regards, > Jakab Haufe (sur5r) > > - -- > ceterum censeo microsoftem esse delendam. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > > iEYEARECAAYFAktXSW8ACgkQ1YAhDic+adbqXACeIJcx6GW6PhSXFO1YF72PafJG > 7t8AoLNwnJYMZ4bssqMZt/nkTIPWs0LI > =vuWN > -----END PGP SIGNATURE----- > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
