Should I open a bug report for this mmnormalize module that does not display anything in the debug logs ?
On Fri, May 11, 2018 at 3:50 PM, Pascal Withopf <[email protected]> wrote: > I created a test for this and everything is working fine with the given > ruleset and message. > > Rsyslog Config: > > module(load="../plugins/imtcp/.libs/imtcp") > module(load="../plugins/mmnormalize/.libs/mmnormalize") > > input(type="imtcp" port="13514" ruleset="norm") > > template(name="outfmt" type="string" string="severity: %$!severity% msg: > %$!msg%\n") > > ruleset(name="norm") { > action(type="mmnormalize" useRawMsg="on" > rulebase="testsuites/mmnormalize_mailinglist_test.rulebase") > action(type="omfile" file="rsyslog.out.log" template="outfmt") > } > > Input: > 2018-05-08 09:30:05.947 CEST [1758] postgres@postgres [local] FATAL: no > pg_hba.conf entry for host "[local]", user "postgres", database "postgres", > SSL off > > Ruleset: > version=2 > rule=: %date:date-iso% %time:word% %tz:word% [%pid:char-to:\x5d%] > %user:char-to:\x40%@%db:word% [%host:char-to:\x5d%] > %severity:char-to:\x3a%: %msg:rest% > > Output: > severity: FATAL msg: no pg_hba.conf entry for host "[local]", user > "postgres", database "postgres", SSL off > > > > 2018-05-08 16:44 GMT+02:00 Flo Rance via rsyslog < > [email protected]> > : > > > Yep, already did that. > > > > On Tue, May 8, 2018 at 4:38 PM, Rainer Gerhards < > [email protected]> > > wrote: > > > > > Doc issue - just remove -c5 > > > > > > Sent from phone, thus brief. > > > > > > Flo Rance <[email protected]> schrieb am Di., 8. Mai 2018, 16:18: > > > > > >> Willingly, but when I try to start the daemon in debug mode, I get the > > >> following error: > > >> > > >> /usr/sbin/rsyslogd -c5 -dn > rsyslog.log > > >> /usr/sbin/rsyslogd: invalid option -- 'c' > > >> usage: rsyslogd [options] > > >> use "man rsyslogd" for details. To run rsyslog interactively, use > > >> "rsyslogd -n" > > >> to run it in debug mode use "rsyslogd -dn" > > >> For further information see http://www.rsyslog.com/doc > > >> > > >> Even if the option is explicitely set in /etc/init.d/rsyslog script: > > >> > > >> RSYSLOGD_OPTIONS="-c5" > > >> > > >> On Tue, May 8, 2018 at 3:21 PM, Rainer Gerhards < > > [email protected] > > >> > wrote: > > >> > > >>> mmhhhh, that's strange. I would ndeed to craft a testbench test. If > > >>> you could create a program debug log, that could probably speed up > the > > >>> solution process. Doc > > >>> https://www.rsyslog.com/doc/v8-stable/troubleshooting/ > > >>> troubleshoot.html#debug-log > > >>> > > >>> Rainer > > >>> > > >>> 2018-05-08 15:05 GMT+02:00 Flo Rance <[email protected]>: > > >>> > No, I did not. It is a very basic rule that I've tested to be sure > > the > > >>> > problem did not come from the more complex one. > > >>> > > > >>> > Here's the message debug output from rsyslog. > > >>> > > > >>> > On Tue, May 8, 2018 at 2:34 PM, Rainer Gerhards < > > >>> [email protected]> > > >>> > wrote: > > >>> >> > > >>> >> I think you picked one wrong file: test.rb I don't know, but the > > >>> >> message debug output from rsyslog is missing. > > >>> >> > > >>> >> Rainer > > >>> >> > > >>> >> 2018-05-08 12:49 GMT+02:00 Flo Rance <[email protected]>: > > >>> >> > Here are the files used to test with lognormalizer. Btw, I'm > using > > >>> >> > rsyslog > > >>> >> > 8.34.0 from the following repo "deb > > >>> >> > http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial main". > > >>> >> > > > >>> >> > On Tue, May 8, 2018 at 11:19 AM, Rainer Gerhards > > >>> >> > <[email protected]> > > >>> >> > wrote: > > >>> >> >> > > >>> >> >> Not sure if I have time, but maybe others have: can you post > both > > >>> the > > >>> >> >> file and the rulebase **AS FILES**? I'll at least try to have a > > >>> look. > > >>> >> >> > > >>> >> >> Rainer > > >>> >> >> > > >>> >> >> 2018-05-08 10:43 GMT+02:00 Flo Rance <[email protected]>: > > >>> >> >> > I've tried that as well. Like that: > > >>> >> >> > > > >>> >> >> > $ cat postgresql.log > > >>> >> >> > 2018-05-08 09:30:05.947 CEST [1758] postgres@postgres > [local] > > >>> FATAL: > > >>> >> >> > no > > >>> >> >> > pg_hba.conf entry for host "[local]", user "postgres", > database > > >>> >> >> > "postgres", > > >>> >> >> > SSL off > > >>> >> >> > > > >>> >> >> > And then: > > >>> >> >> > > > >>> >> >> > $ /usr/lib/lognorm/lognormalizer -p -r > > >>> /home/syslog/rules/test.rb < > > >>> >> >> > postgresql.log > > >>> >> >> > { "msg": "2018-05-08 09:30:05.947 CEST [1758] > postgres@postgres > > >>> >> >> > [local] > > >>> >> >> > FATAL: no pg_hba.conf entry for host \"[local]\", user > > >>> \"postgres\", > > >>> >> >> > database \"postgres\", SSL off" } > > >>> >> >> > > > >>> >> >> > Honestly, I can't figure out what's wrong. > > >>> >> >> > > > >>> >> >> > On Tue, May 8, 2018 at 9:55 AM, Rainer Gerhards > > >>> >> >> > <[email protected]> > > >>> >> >> > wrote: > > >>> >> >> >> > > >>> >> >> >> I am not sure if echo gives you exactly what you think it > > does. > > >>> Most > > >>> >> >> >> importantly, I am not sure if it skips the (important) first > > >>> space. > > >>> >> >> >> > > >>> >> >> >> I usually place the data into a file, so I know exactly what > > it > > >>> is. > > >>> >> >> >> Then I use stdin redirection (<) to put that file into the > > >>> program, > > >>> >> >> >> e.g. > > >>> >> >> >> > > >>> >> >> >> $ loganalyzer < inputfile .... > > >>> >> >> >> > > >>> >> >> >> HTH > > >>> >> >> >> Rainer > > >>> >> >> >> > > >>> >> >> >> 2018-05-08 9:42 GMT+02:00 Flo Rance <[email protected]>: > > >>> >> >> >> > May anyone give me any tips ? Because I'm completely > stucked > > >>> >> >> >> > there. > > >>> >> >> >> > > > >>> >> >> >> > I've tried again with a very simple rule: > > >>> >> >> >> > > > >>> >> >> >> > version=2 > > >>> >> >> >> > rule=: %msg:rest% > > >>> >> >> >> > > > >>> >> >> >> > It's working correctly with lognormalizer: > > >>> >> >> >> > > > >>> >> >> >> > echo ' 2018-05-08 09:30:05.947 CEST [1758] > postgres@postgres > > >>> >> >> >> > [local] > > >>> >> >> >> > FATAL: > > >>> >> >> >> > no pg_hba.conf entry for host "[local]", user "postgres", > > >>> database > > >>> >> >> >> > "postgres", SSL off' | /usr/lib/lognorm/lognormalizer -p > -r > > >>> >> >> >> > /home/syslog/rules/test.rb > > >>> >> >> >> > { "msg": "2018-05-08 09:30:05.947 CEST [1758] > > >>> postgres@postgres > > >>> >> >> >> > [local] > > >>> >> >> >> > FATAL: no pg_hba.conf entry for host \"[local]\", user > > >>> >> >> >> > \"postgres\", > > >>> >> >> >> > database \"postgres\", SSL off" } > > >>> >> >> >> > > > >>> >> >> >> > But I still get nothing in the debug logs: > > >>> >> >> >> > > > >>> >> >> >> > Debug line with all properties: > > >>> >> >> >> > FROMHOST: 'sc006692.aevisintra.ch', fromhost-ip: > > '127.0.0.1', > > >>> >> >> >> > HOSTNAME: > > >>> >> >> >> > 'sc006692.aevisintra.ch', PRI: 155, > > >>> >> >> >> > syslogtag 'docker_fluance-authenticationdb[1116]:', > > >>> programname: > > >>> >> >> >> > 'docker_fluance-authenticationdb', APP-NAME: > > >>> >> >> >> > 'docker_fluance-authenticationdb', PROCID: '1116', MSGID: > > >>> '-', > > >>> >> >> >> > TIMESTAMP: 'May 8 09:30:05', STRUCTURED-DATA: '-', > > >>> >> >> >> > msg: ' 2018-05-08 09:30:05.947 CEST [1758] > postgres@postgres > > >>> >> >> >> > [local] > > >>> >> >> >> > FATAL: > > >>> >> >> >> > no pg_hba.conf entry for host "[local]", user "postgres", > > >>> database > > >>> >> >> >> > "postgres", SSL off' > > >>> >> >> >> > escaped msg: ' 2018-05-08 09:30:05.947 CEST [1758] > > >>> >> >> >> > postgres@postgres > > >>> >> >> >> > [local] > > >>> >> >> >> > FATAL: no pg_hba.conf entry for host "[local]", user > > >>> "postgres", > > >>> >> >> >> > database > > >>> >> >> >> > "postgres", SSL off' > > >>> >> >> >> > inputname: imuxsock rawmsg: '<155>May 8 09:30:05 > > >>> >> >> >> > docker_fluance-authenticationdb[1116]: 2018-05-08 > > >>> 09:30:05.947 > > >>> >> >> >> > CEST > > >>> >> >> >> > [1758] > > >>> >> >> >> > postgres@postgres [local] FATAL: no pg_hba.conf entry > for > > >>> host > > >>> >> >> >> > "[local]", > > >>> >> >> >> > user "postgres", database "postgres", SSL off' > > >>> >> >> >> > $!: > > >>> >> >> >> > $.: > > >>> >> >> >> > $/: > > >>> >> >> >> > > > >>> >> >> >> > On Fri, May 4, 2018 at 10:35 AM, Flo Rance < > > >>> [email protected]> > > >>> >> >> >> > wrote: > > >>> >> >> >> >> > > >>> >> >> >> >> I don't really understand what you mean by adding the > > >>> appropriate > > >>> >> >> >> >> quotes > > >>> >> >> >> >> to the rule. They are part of the %msg:rest% at the end. > > >>> >> >> >> >> > > >>> >> >> >> >> I've tried again with the single quoted message and I get > > the > > >>> >> >> >> >> following > > >>> >> >> >> >> result: > > >>> >> >> >> >> > > >>> >> >> >> >> echo ' 2018-05-04 10:24:16.573 CEST [53] > postgres@postgres > > >>> >> >> >> >> [local] > > >>> >> >> >> >> FATAL: > > >>> >> >> >> >> no pg_hba.conf entry for host "[local]", user "postgres", > > >>> >> >> >> >> database > > >>> >> >> >> >> "postgres", SSL off' | /usr/lib/lognorm/lognormalizer -r > > >>> >> >> >> >> /home/syslog/rules/postgresql.rb > > >>> >> >> >> >> { "msg": " no pg_hba.conf entry for host \"[local]\", > user > > >>> >> >> >> >> \"postgres\", > > >>> >> >> >> >> database \"postgres\", SSL off", "severity": "FATAL", > > "host": > > >>> >> >> >> >> "local", > > >>> >> >> >> >> "db": > > >>> >> >> >> >> "postgres", "user": "postgres", "pid": "53", "tz": > "CEST", > > >>> >> >> >> >> "time": > > >>> >> >> >> >> "10:24:16.573", "date": "2018-05-04" } > > >>> >> >> >> >> > > >>> >> >> >> >> But there's still nothing in the debug logs: > > >>> >> >> >> >> > > >>> >> >> >> >> Debug line with all properties: > > >>> >> >> >> >> FROMHOST: 'sc005827.domain', fromhost-ip: '127.0.0.1', > > >>> HOSTNAME: > > >>> >> >> >> >> 'sc005827.domain', PRI: 155, > > >>> >> >> >> >> syslogtag 'docker_fluance-authenticationdb[1092]:', > > >>> programname: > > >>> >> >> >> >> 'docker_fluance-authenticationdb', APP-NAME: > > >>> >> >> >> >> 'docker_fluance-authenticationdb', PROCID: '1092', > MSGID: > > >>> '-', > > >>> >> >> >> >> TIMESTAMP: 'May 4 10:24:16', STRUCTURED-DATA: '-', > > >>> >> >> >> >> msg: ' 2018-05-04 10:24:16.573 CEST [53] > postgres@postgres > > >>> >> >> >> >> [local] > > >>> >> >> >> >> FATAL: > > >>> >> >> >> >> no pg_hba.conf entry for host "[local]", user "postgres", > > >>> >> >> >> >> database > > >>> >> >> >> >> "postgres", SSL off' > > >>> >> >> >> >> escaped msg: ' 2018-05-04 10:24:16.573 CEST [53] > > >>> >> >> >> >> postgres@postgres > > >>> >> >> >> >> [local] > > >>> >> >> >> >> FATAL: no pg_hba.conf entry for host "[local]", user > > >>> "postgres", > > >>> >> >> >> >> database > > >>> >> >> >> >> "postgres", SSL off' > > >>> >> >> >> >> inputname: imuxsock rawmsg: '<155>May 4 10:24:16 > > >>> >> >> >> >> docker_fluance-authenticationdb[1092]: 2018-05-04 > > >>> 10:24:16.573 > > >>> >> >> >> >> CEST > > >>> >> >> >> >> [53] > > >>> >> >> >> >> postgres@postgres [local] FATAL: no pg_hba.conf entry > for > > >>> host > > >>> >> >> >> >> "[local]", > > >>> >> >> >> >> user "postgres", database "postgres", SSL off' > > >>> >> >> >> >> $!: > > >>> >> >> >> >> $.: > > >>> >> >> >> >> $/: > > >>> >> >> >> >> > > >>> >> >> >> >> On Thu, May 3, 2018 at 11:38 PM, David Lang < > [email protected] > > > > > >>> >> >> >> >> wrote: > > >>> >> >> >> >>> > > >>> >> >> >> >>> your ruleset doesn't have quotes in it. > > >>> >> >> >> >>> > > >>> >> >> >> >>> the way you are echoing it with the wrong quotes is > > causing > > >>> >> >> >> >>> things > > >>> >> >> >> >>> to > > >>> >> >> >> >>> not > > >>> >> >> >> >>> work as expected. the line that you are actually > > processing > > >>> is: > > >>> >> >> >> >>> > > >>> >> >> >> >>> 2018-05-03 11:04:20.201 CEST [24873] postgres@postgres > > >>> [local] > > >>> >> >> >> >>> FATAL: > > >>> >> >> >> >>> no pg_hba.conf entry for host [local], user postgres, > > >>> database > > >>> >> >> >> >>> postgres, SSL > > >>> >> >> >> >>> off > > >>> >> >> >> >>> > > >>> >> >> >> >>> while what is being sent is actually: > > >>> >> >> >> >>> > > >>> >> >> >> >>> 2018-05-03 11:04:20.201 CEST [24873] postgres@postgres > > >>> [local] > > >>> >> >> >> >>> FATAL: > > >>> >> >> >> >>> no pg_hba.conf entry for host "[local]", user > "postgres", > > >>> >> >> >> >>> database > > >>> >> >> >> >>> "postgres", SSL off > > >>> >> >> >> >>> > > >>> >> >> >> >>> add the appropriate quotes to your rule and it should > work > > >>> >> >> >> >>> properly > > >>> >> >> >> >>> > > >>> >> >> >> >>> David Lang > > >>> >> >> >> >>> > > >>> >> >> >> >>> On Thu, 3 May 2018, Flo Rance wrote: > > >>> >> >> >> >>> > > >>> >> >> >> >>>> Date: Thu, 3 May 2018 11:13:02 +0200 > > >>> >> >> >> >>>> From: Flo Rance <[email protected]> > > >>> >> >> >> >>>> To: David Lang <[email protected]> > > >>> >> >> >> >>>> Cc: Rainer Gerhards <[email protected]>, > > >>> >> >> >> >>>> rsyslog-users <[email protected]> > > >>> >> >> >> >>>> Subject: Re: [rsyslog] Filter on priority from docker > > >>> >> >> >> >>>> > > >>> >> >> >> >>>> Here's the rule that is applied: > > >>> >> >> >> >>>> > > >>> >> >> >> >>>> rule=: %date:date-iso% %time:word% %tz:word% > > >>> >> >> >> >>>> [%pid:char-to:\x5d%] > > >>> >> >> >> >>>> %user:char-to:\x40%@%db:word% [%host:char-to:\x5d%] > > >>> >> >> >> >>>> %severity:char-to:\x3a%: %msg:rest% > > >>> >> >> >> >>>> > > >>> >> >> >> >>>> Here's the output of the lognormalizer utility: > > >>> >> >> >> >>>> > > >>> >> >> >> >>>> echo " 2018-05-03 11:04:20.201 CEST [24873] > > >>> postgres@postgres > > >>> >> >> >> >>>> [local] > > >>> >> >> >> >>>> FATAL: no pg_hba.conf entry for host "[local]", user > > >>> >> >> >> >>>> "postgres", > > >>> >> >> >> >>>> database > > >>> >> >> >> >>>> "postgres", SSL off" | /usr/lib/lognorm/lognormalizer > -r > > >>> >> >> >> >>>> /home/syslog/rules/postgresql.rb > > >>> >> >> >> >>>> { "msg": " no pg_hba.conf entry for host [local], user > > >>> >> >> >> >>>> postgres, > > >>> >> >> >> >>>> database > > >>> >> >> >> >>>> postgres, SSL off", "severity": "FATAL", "host": > "local", > > >>> "db": > > >>> >> >> >> >>>> "postgres", > > >>> >> >> >> >>>> "user": "postgres", "pid": "24873", "tz": "CEST", > "time": > > >>> >> >> >> >>>> "11:04:20.201", > > >>> >> >> >> >>>> "date": "2018-05-03" } > > >>> >> >> >> >>>> > > >>> >> >> >> >>>> and finally the output of rsyslog debug: > > >>> >> >> >> >>>> > > >>> >> >> >> >>>> Debug line with all properties: > > >>> >> >> >> >>>> FROMHOST: 'sc006692.domain', fromhost-ip: '127.0.0.1', > > >>> >> >> >> >>>> HOSTNAME: > > >>> >> >> >> >>>> 'sc006692.domain', PRI: 155, > > >>> >> >> >> >>>> syslogtag 'docker_fluance-ehealthdb[1116]:', > > programname: > > >>> >> >> >> >>>> 'docker_fluance-ehealthdb', APP-NAME: > > >>> >> >> >> >>>> 'docker_fluance-ehealthdb', > > >>> >> >> >> >>>> PROCID: > > >>> >> >> >> >>>> '1116', MSGID: '-', > > >>> >> >> >> >>>> TIMESTAMP: 'May 3 11:04:20', STRUCTURED-DATA: '-', > > >>> >> >> >> >>>> msg: ' 2018-05-03 11:04:20.201 CEST [24873] > > >>> postgres@postgres > > >>> >> >> >> >>>> [local] > > >>> >> >> >> >>>> FATAL: no pg_hba.conf entry for host "[local]", user > > >>> >> >> >> >>>> "postgres", > > >>> >> >> >> >>>> database > > >>> >> >> >> >>>> "postgres", SSL off' > > >>> >> >> >> >>>> escaped msg: ' 2018-05-03 11:04:20.201 CEST [24873] > > >>> >> >> >> >>>> postgres@postgres > > >>> >> >> >> >>>> [local] FATAL: no pg_hba.conf entry for host > "[local]", > > >>> user > > >>> >> >> >> >>>> "postgres", > > >>> >> >> >> >>>> database "postgres", SSL off' > > >>> >> >> >> >>>> inputname: imuxsock rawmsg: '<155>May 3 11:04:20 > > >>> >> >> >> >>>> docker_fluance-ehealthdb[1116]: 2018-05-03 > 11:04:20.201 > > >>> CEST > > >>> >> >> >> >>>> [24873] > > >>> >> >> >> >>>> postgres@postgres [local] FATAL: no pg_hba.conf entry > > >>> for host > > >>> >> >> >> >>>> "[local]", > > >>> >> >> >> >>>> user "postgres", database "postgres", SSL off' > > >>> >> >> >> >>>> $!: > > >>> >> >> >> >>>> $.: > > >>> >> >> >> >>>> $/: > > >>> >> >> >> >>>> > > >>> >> >> >> >>>> On Wed, May 2, 2018 at 11:20 PM, David Lang < > > [email protected] > > >>> > > > >>> >> >> >> >>>> wrote: > > >>> >> >> >> >>>> > > >>> >> >> >> >>>>> Please post your rulebase and the output from > > >>> >> >> >> >>>>> RSYSLOG_DebugFormat > > >>> >> >> >> >>>>> so > > >>> >> >> >> >>>>> that > > >>> >> >> >> >>>>> we can look at a message that should be matched and > what > > >>> the > > >>> >> >> >> >>>>> ruleset > > >>> >> >> >> >>>>> for > > >>> >> >> >> >>>>> the match is. > > >>> >> >> >> >>>>> > > >>> >> >> >> >>>>> Odds are that there is something different in the > > message > > >>> than > > >>> >> >> >> >>>>> you > > >>> >> >> >> >>>>> think > > >>> >> >> >> >>>>> it is, so your rule doesn't actually match. > > >>> >> >> >> >>>>> > > >>> >> >> >> >>>> > > >>> >> >> >> >> > > >>> >> >> >> > > > >>> >> >> > > > >>> >> >> > > > >>> >> > > > >>> >> > > > >>> > > > >>> > > > >>> > > >> > > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
