I created a test for this and everything is working fine with the given
ruleset and message.
Rsyslog Config:
module(load="../plugins/imtcp/.libs/imtcp")
module(load="../plugins/mmnormalize/.libs/mmnormalize")
input(type="imtcp" port="13514" ruleset="norm")
template(name="outfmt" type="string" string="severity: %$!severity% msg:
%$!msg%\n")
ruleset(name="norm") {
action(type="mmnormalize" useRawMsg="on"
rulebase="testsuites/mmnormalize_mailinglist_test.rulebase")
action(type="omfile" file="rsyslog.out.log" template="outfmt")
}
Input:
2018-05-08 09:30:05.947 CEST [1758] postgres@postgres [local] FATAL: no
pg_hba.conf entry for host "[local]", user "postgres", database "postgres",
SSL off
Ruleset:
version=2
rule=: %date:date-iso% %time:word% %tz:word% [%pid:char-to:\x5d%]
%user:char-to:\x40%@%db:word% [%host:char-to:\x5d%]
%severity:char-to:\x3a%: %msg:rest%
Output:
severity: FATAL msg: no pg_hba.conf entry for host "[local]", user
"postgres", database "postgres", SSL off
2018-05-08 16:44 GMT+02:00 Flo Rance via rsyslog <[email protected]>
:
> Yep, already did that.
>
> On Tue, May 8, 2018 at 4:38 PM, Rainer Gerhards <[email protected]>
> wrote:
>
> > Doc issue - just remove -c5
> >
> > Sent from phone, thus brief.
> >
> > Flo Rance <[email protected]> schrieb am Di., 8. Mai 2018, 16:18:
> >
> >> Willingly, but when I try to start the daemon in debug mode, I get the
> >> following error:
> >>
> >> /usr/sbin/rsyslogd -c5 -dn > rsyslog.log
> >> /usr/sbin/rsyslogd: invalid option -- 'c'
> >> usage: rsyslogd [options]
> >> use "man rsyslogd" for details. To run rsyslog interactively, use
> >> "rsyslogd -n"
> >> to run it in debug mode use "rsyslogd -dn"
> >> For further information see http://www.rsyslog.com/doc
> >>
> >> Even if the option is explicitely set in /etc/init.d/rsyslog script:
> >>
> >> RSYSLOGD_OPTIONS="-c5"
> >>
> >> On Tue, May 8, 2018 at 3:21 PM, Rainer Gerhards <
> [email protected]
> >> > wrote:
> >>
> >>> mmhhhh, that's strange. I would ndeed to craft a testbench test. If
> >>> you could create a program debug log, that could probably speed up the
> >>> solution process. Doc
> >>> https://www.rsyslog.com/doc/v8-stable/troubleshooting/
> >>> troubleshoot.html#debug-log
> >>>
> >>> Rainer
> >>>
> >>> 2018-05-08 15:05 GMT+02:00 Flo Rance <[email protected]>:
> >>> > No, I did not. It is a very basic rule that I've tested to be sure
> the
> >>> > problem did not come from the more complex one.
> >>> >
> >>> > Here's the message debug output from rsyslog.
> >>> >
> >>> > On Tue, May 8, 2018 at 2:34 PM, Rainer Gerhards <
> >>> [email protected]>
> >>> > wrote:
> >>> >>
> >>> >> I think you picked one wrong file: test.rb I don't know, but the
> >>> >> message debug output from rsyslog is missing.
> >>> >>
> >>> >> Rainer
> >>> >>
> >>> >> 2018-05-08 12:49 GMT+02:00 Flo Rance <[email protected]>:
> >>> >> > Here are the files used to test with lognormalizer. Btw, I'm using
> >>> >> > rsyslog
> >>> >> > 8.34.0 from the following repo "deb
> >>> >> > http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial main".
> >>> >> >
> >>> >> > On Tue, May 8, 2018 at 11:19 AM, Rainer Gerhards
> >>> >> > <[email protected]>
> >>> >> > wrote:
> >>> >> >>
> >>> >> >> Not sure if I have time, but maybe others have: can you post both
> >>> the
> >>> >> >> file and the rulebase **AS FILES**? I'll at least try to have a
> >>> look.
> >>> >> >>
> >>> >> >> Rainer
> >>> >> >>
> >>> >> >> 2018-05-08 10:43 GMT+02:00 Flo Rance <[email protected]>:
> >>> >> >> > I've tried that as well. Like that:
> >>> >> >> >
> >>> >> >> > $ cat postgresql.log
> >>> >> >> > 2018-05-08 09:30:05.947 CEST [1758] postgres@postgres [local]
> >>> FATAL:
> >>> >> >> > no
> >>> >> >> > pg_hba.conf entry for host "[local]", user "postgres", database
> >>> >> >> > "postgres",
> >>> >> >> > SSL off
> >>> >> >> >
> >>> >> >> > And then:
> >>> >> >> >
> >>> >> >> > $ /usr/lib/lognorm/lognormalizer -p -r
> >>> /home/syslog/rules/test.rb <
> >>> >> >> > postgresql.log
> >>> >> >> > { "msg": "2018-05-08 09:30:05.947 CEST [1758] postgres@postgres
> >>> >> >> > [local]
> >>> >> >> > FATAL: no pg_hba.conf entry for host \"[local]\", user
> >>> \"postgres\",
> >>> >> >> > database \"postgres\", SSL off" }
> >>> >> >> >
> >>> >> >> > Honestly, I can't figure out what's wrong.
> >>> >> >> >
> >>> >> >> > On Tue, May 8, 2018 at 9:55 AM, Rainer Gerhards
> >>> >> >> > <[email protected]>
> >>> >> >> > wrote:
> >>> >> >> >>
> >>> >> >> >> I am not sure if echo gives you exactly what you think it
> does.
> >>> Most
> >>> >> >> >> importantly, I am not sure if it skips the (important) first
> >>> space.
> >>> >> >> >>
> >>> >> >> >> I usually place the data into a file, so I know exactly what
> it
> >>> is.
> >>> >> >> >> Then I use stdin redirection (<) to put that file into the
> >>> program,
> >>> >> >> >> e.g.
> >>> >> >> >>
> >>> >> >> >> $ loganalyzer < inputfile ....
> >>> >> >> >>
> >>> >> >> >> HTH
> >>> >> >> >> Rainer
> >>> >> >> >>
> >>> >> >> >> 2018-05-08 9:42 GMT+02:00 Flo Rance <[email protected]>:
> >>> >> >> >> > May anyone give me any tips ? Because I'm completely stucked
> >>> >> >> >> > there.
> >>> >> >> >> >
> >>> >> >> >> > I've tried again with a very simple rule:
> >>> >> >> >> >
> >>> >> >> >> > version=2
> >>> >> >> >> > rule=: %msg:rest%
> >>> >> >> >> >
> >>> >> >> >> > It's working correctly with lognormalizer:
> >>> >> >> >> >
> >>> >> >> >> > echo ' 2018-05-08 09:30:05.947 CEST [1758] postgres@postgres
> >>> >> >> >> > [local]
> >>> >> >> >> > FATAL:
> >>> >> >> >> > no pg_hba.conf entry for host "[local]", user "postgres",
> >>> database
> >>> >> >> >> > "postgres", SSL off' | /usr/lib/lognorm/lognormalizer -p -r
> >>> >> >> >> > /home/syslog/rules/test.rb
> >>> >> >> >> > { "msg": "2018-05-08 09:30:05.947 CEST [1758]
> >>> postgres@postgres
> >>> >> >> >> > [local]
> >>> >> >> >> > FATAL: no pg_hba.conf entry for host \"[local]\", user
> >>> >> >> >> > \"postgres\",
> >>> >> >> >> > database \"postgres\", SSL off" }
> >>> >> >> >> >
> >>> >> >> >> > But I still get nothing in the debug logs:
> >>> >> >> >> >
> >>> >> >> >> > Debug line with all properties:
> >>> >> >> >> > FROMHOST: 'sc006692.aevisintra.ch', fromhost-ip:
> '127.0.0.1',
> >>> >> >> >> > HOSTNAME:
> >>> >> >> >> > 'sc006692.aevisintra.ch', PRI: 155,
> >>> >> >> >> > syslogtag 'docker_fluance-authenticationdb[1116]:',
> >>> programname:
> >>> >> >> >> > 'docker_fluance-authenticationdb', APP-NAME:
> >>> >> >> >> > 'docker_fluance-authenticationdb', PROCID: '1116', MSGID:
> >>> '-',
> >>> >> >> >> > TIMESTAMP: 'May 8 09:30:05', STRUCTURED-DATA: '-',
> >>> >> >> >> > msg: ' 2018-05-08 09:30:05.947 CEST [1758] postgres@postgres
> >>> >> >> >> > [local]
> >>> >> >> >> > FATAL:
> >>> >> >> >> > no pg_hba.conf entry for host "[local]", user "postgres",
> >>> database
> >>> >> >> >> > "postgres", SSL off'
> >>> >> >> >> > escaped msg: ' 2018-05-08 09:30:05.947 CEST [1758]
> >>> >> >> >> > postgres@postgres
> >>> >> >> >> > [local]
> >>> >> >> >> > FATAL: no pg_hba.conf entry for host "[local]", user
> >>> "postgres",
> >>> >> >> >> > database
> >>> >> >> >> > "postgres", SSL off'
> >>> >> >> >> > inputname: imuxsock rawmsg: '<155>May 8 09:30:05
> >>> >> >> >> > docker_fluance-authenticationdb[1116]: 2018-05-08
> >>> 09:30:05.947
> >>> >> >> >> > CEST
> >>> >> >> >> > [1758]
> >>> >> >> >> > postgres@postgres [local] FATAL: no pg_hba.conf entry for
> >>> host
> >>> >> >> >> > "[local]",
> >>> >> >> >> > user "postgres", database "postgres", SSL off'
> >>> >> >> >> > $!:
> >>> >> >> >> > $.:
> >>> >> >> >> > $/:
> >>> >> >> >> >
> >>> >> >> >> > On Fri, May 4, 2018 at 10:35 AM, Flo Rance <
> >>> [email protected]>
> >>> >> >> >> > wrote:
> >>> >> >> >> >>
> >>> >> >> >> >> I don't really understand what you mean by adding the
> >>> appropriate
> >>> >> >> >> >> quotes
> >>> >> >> >> >> to the rule. They are part of the %msg:rest% at the end.
> >>> >> >> >> >>
> >>> >> >> >> >> I've tried again with the single quoted message and I get
> the
> >>> >> >> >> >> following
> >>> >> >> >> >> result:
> >>> >> >> >> >>
> >>> >> >> >> >> echo ' 2018-05-04 10:24:16.573 CEST [53] postgres@postgres
> >>> >> >> >> >> [local]
> >>> >> >> >> >> FATAL:
> >>> >> >> >> >> no pg_hba.conf entry for host "[local]", user "postgres",
> >>> >> >> >> >> database
> >>> >> >> >> >> "postgres", SSL off' | /usr/lib/lognorm/lognormalizer -r
> >>> >> >> >> >> /home/syslog/rules/postgresql.rb
> >>> >> >> >> >> { "msg": " no pg_hba.conf entry for host \"[local]\", user
> >>> >> >> >> >> \"postgres\",
> >>> >> >> >> >> database \"postgres\", SSL off", "severity": "FATAL",
> "host":
> >>> >> >> >> >> "local",
> >>> >> >> >> >> "db":
> >>> >> >> >> >> "postgres", "user": "postgres", "pid": "53", "tz": "CEST",
> >>> >> >> >> >> "time":
> >>> >> >> >> >> "10:24:16.573", "date": "2018-05-04" }
> >>> >> >> >> >>
> >>> >> >> >> >> But there's still nothing in the debug logs:
> >>> >> >> >> >>
> >>> >> >> >> >> Debug line with all properties:
> >>> >> >> >> >> FROMHOST: 'sc005827.domain', fromhost-ip: '127.0.0.1',
> >>> HOSTNAME:
> >>> >> >> >> >> 'sc005827.domain', PRI: 155,
> >>> >> >> >> >> syslogtag 'docker_fluance-authenticationdb[1092]:',
> >>> programname:
> >>> >> >> >> >> 'docker_fluance-authenticationdb', APP-NAME:
> >>> >> >> >> >> 'docker_fluance-authenticationdb', PROCID: '1092', MSGID:
> >>> '-',
> >>> >> >> >> >> TIMESTAMP: 'May 4 10:24:16', STRUCTURED-DATA: '-',
> >>> >> >> >> >> msg: ' 2018-05-04 10:24:16.573 CEST [53] postgres@postgres
> >>> >> >> >> >> [local]
> >>> >> >> >> >> FATAL:
> >>> >> >> >> >> no pg_hba.conf entry for host "[local]", user "postgres",
> >>> >> >> >> >> database
> >>> >> >> >> >> "postgres", SSL off'
> >>> >> >> >> >> escaped msg: ' 2018-05-04 10:24:16.573 CEST [53]
> >>> >> >> >> >> postgres@postgres
> >>> >> >> >> >> [local]
> >>> >> >> >> >> FATAL: no pg_hba.conf entry for host "[local]", user
> >>> "postgres",
> >>> >> >> >> >> database
> >>> >> >> >> >> "postgres", SSL off'
> >>> >> >> >> >> inputname: imuxsock rawmsg: '<155>May 4 10:24:16
> >>> >> >> >> >> docker_fluance-authenticationdb[1092]: 2018-05-04
> >>> 10:24:16.573
> >>> >> >> >> >> CEST
> >>> >> >> >> >> [53]
> >>> >> >> >> >> postgres@postgres [local] FATAL: no pg_hba.conf entry for
> >>> host
> >>> >> >> >> >> "[local]",
> >>> >> >> >> >> user "postgres", database "postgres", SSL off'
> >>> >> >> >> >> $!:
> >>> >> >> >> >> $.:
> >>> >> >> >> >> $/:
> >>> >> >> >> >>
> >>> >> >> >> >> On Thu, May 3, 2018 at 11:38 PM, David Lang <[email protected]
> >
> >>> >> >> >> >> wrote:
> >>> >> >> >> >>>
> >>> >> >> >> >>> your ruleset doesn't have quotes in it.
> >>> >> >> >> >>>
> >>> >> >> >> >>> the way you are echoing it with the wrong quotes is
> causing
> >>> >> >> >> >>> things
> >>> >> >> >> >>> to
> >>> >> >> >> >>> not
> >>> >> >> >> >>> work as expected. the line that you are actually
> processing
> >>> is:
> >>> >> >> >> >>>
> >>> >> >> >> >>> 2018-05-03 11:04:20.201 CEST [24873] postgres@postgres
> >>> [local]
> >>> >> >> >> >>> FATAL:
> >>> >> >> >> >>> no pg_hba.conf entry for host [local], user postgres,
> >>> database
> >>> >> >> >> >>> postgres, SSL
> >>> >> >> >> >>> off
> >>> >> >> >> >>>
> >>> >> >> >> >>> while what is being sent is actually:
> >>> >> >> >> >>>
> >>> >> >> >> >>> 2018-05-03 11:04:20.201 CEST [24873] postgres@postgres
> >>> [local]
> >>> >> >> >> >>> FATAL:
> >>> >> >> >> >>> no pg_hba.conf entry for host "[local]", user "postgres",
> >>> >> >> >> >>> database
> >>> >> >> >> >>> "postgres", SSL off
> >>> >> >> >> >>>
> >>> >> >> >> >>> add the appropriate quotes to your rule and it should work
> >>> >> >> >> >>> properly
> >>> >> >> >> >>>
> >>> >> >> >> >>> David Lang
> >>> >> >> >> >>>
> >>> >> >> >> >>> On Thu, 3 May 2018, Flo Rance wrote:
> >>> >> >> >> >>>
> >>> >> >> >> >>>> Date: Thu, 3 May 2018 11:13:02 +0200
> >>> >> >> >> >>>> From: Flo Rance <[email protected]>
> >>> >> >> >> >>>> To: David Lang <[email protected]>
> >>> >> >> >> >>>> Cc: Rainer Gerhards <[email protected]>,
> >>> >> >> >> >>>> rsyslog-users <[email protected]>
> >>> >> >> >> >>>> Subject: Re: [rsyslog] Filter on priority from docker
> >>> >> >> >> >>>>
> >>> >> >> >> >>>> Here's the rule that is applied:
> >>> >> >> >> >>>>
> >>> >> >> >> >>>> rule=: %date:date-iso% %time:word% %tz:word%
> >>> >> >> >> >>>> [%pid:char-to:\x5d%]
> >>> >> >> >> >>>> %user:char-to:\x40%@%db:word% [%host:char-to:\x5d%]
> >>> >> >> >> >>>> %severity:char-to:\x3a%: %msg:rest%
> >>> >> >> >> >>>>
> >>> >> >> >> >>>> Here's the output of the lognormalizer utility:
> >>> >> >> >> >>>>
> >>> >> >> >> >>>> echo " 2018-05-03 11:04:20.201 CEST [24873]
> >>> postgres@postgres
> >>> >> >> >> >>>> [local]
> >>> >> >> >> >>>> FATAL: no pg_hba.conf entry for host "[local]", user
> >>> >> >> >> >>>> "postgres",
> >>> >> >> >> >>>> database
> >>> >> >> >> >>>> "postgres", SSL off" | /usr/lib/lognorm/lognormalizer -r
> >>> >> >> >> >>>> /home/syslog/rules/postgresql.rb
> >>> >> >> >> >>>> { "msg": " no pg_hba.conf entry for host [local], user
> >>> >> >> >> >>>> postgres,
> >>> >> >> >> >>>> database
> >>> >> >> >> >>>> postgres, SSL off", "severity": "FATAL", "host": "local",
> >>> "db":
> >>> >> >> >> >>>> "postgres",
> >>> >> >> >> >>>> "user": "postgres", "pid": "24873", "tz": "CEST", "time":
> >>> >> >> >> >>>> "11:04:20.201",
> >>> >> >> >> >>>> "date": "2018-05-03" }
> >>> >> >> >> >>>>
> >>> >> >> >> >>>> and finally the output of rsyslog debug:
> >>> >> >> >> >>>>
> >>> >> >> >> >>>> Debug line with all properties:
> >>> >> >> >> >>>> FROMHOST: 'sc006692.domain', fromhost-ip: '127.0.0.1',
> >>> >> >> >> >>>> HOSTNAME:
> >>> >> >> >> >>>> 'sc006692.domain', PRI: 155,
> >>> >> >> >> >>>> syslogtag 'docker_fluance-ehealthdb[1116]:',
> programname:
> >>> >> >> >> >>>> 'docker_fluance-ehealthdb', APP-NAME:
> >>> >> >> >> >>>> 'docker_fluance-ehealthdb',
> >>> >> >> >> >>>> PROCID:
> >>> >> >> >> >>>> '1116', MSGID: '-',
> >>> >> >> >> >>>> TIMESTAMP: 'May 3 11:04:20', STRUCTURED-DATA: '-',
> >>> >> >> >> >>>> msg: ' 2018-05-03 11:04:20.201 CEST [24873]
> >>> postgres@postgres
> >>> >> >> >> >>>> [local]
> >>> >> >> >> >>>> FATAL: no pg_hba.conf entry for host "[local]", user
> >>> >> >> >> >>>> "postgres",
> >>> >> >> >> >>>> database
> >>> >> >> >> >>>> "postgres", SSL off'
> >>> >> >> >> >>>> escaped msg: ' 2018-05-03 11:04:20.201 CEST [24873]
> >>> >> >> >> >>>> postgres@postgres
> >>> >> >> >> >>>> [local] FATAL: no pg_hba.conf entry for host "[local]",
> >>> user
> >>> >> >> >> >>>> "postgres",
> >>> >> >> >> >>>> database "postgres", SSL off'
> >>> >> >> >> >>>> inputname: imuxsock rawmsg: '<155>May 3 11:04:20
> >>> >> >> >> >>>> docker_fluance-ehealthdb[1116]: 2018-05-03 11:04:20.201
> >>> CEST
> >>> >> >> >> >>>> [24873]
> >>> >> >> >> >>>> postgres@postgres [local] FATAL: no pg_hba.conf entry
> >>> for host
> >>> >> >> >> >>>> "[local]",
> >>> >> >> >> >>>> user "postgres", database "postgres", SSL off'
> >>> >> >> >> >>>> $!:
> >>> >> >> >> >>>> $.:
> >>> >> >> >> >>>> $/:
> >>> >> >> >> >>>>
> >>> >> >> >> >>>> On Wed, May 2, 2018 at 11:20 PM, David Lang <
> [email protected]
> >>> >
> >>> >> >> >> >>>> wrote:
> >>> >> >> >> >>>>
> >>> >> >> >> >>>>> Please post your rulebase and the output from
> >>> >> >> >> >>>>> RSYSLOG_DebugFormat
> >>> >> >> >> >>>>> so
> >>> >> >> >> >>>>> that
> >>> >> >> >> >>>>> we can look at a message that should be matched and what
> >>> the
> >>> >> >> >> >>>>> ruleset
> >>> >> >> >> >>>>> for
> >>> >> >> >> >>>>> the match is.
> >>> >> >> >> >>>>>
> >>> >> >> >> >>>>> Odds are that there is something different in the
> message
> >>> than
> >>> >> >> >> >>>>> you
> >>> >> >> >> >>>>> think
> >>> >> >> >> >>>>> it is, so your rule doesn't actually match.
> >>> >> >> >> >>>>>
> >>> >> >> >> >>>>
> >>> >> >> >> >>
> >>> >> >> >> >
> >>> >> >> >
> >>> >> >> >
> >>> >> >
> >>> >> >
> >>> >
> >>> >
> >>>
> >>
> >>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
