Yes, I did and I was somehow lost in the section "Rules". http://www.liblognorm.com/files/manual/configuration.html#rulebase
Concretely, my idea is to normalize the msg part: msg: ' 2018-04-26 10:22:45.283 CEST [13518] postgres@postgres [local] FATAL: no pg_hba.conf entry for host "[local]", user "postgres", database "postgres", SSL off' However, not so much examples treat about severity. On Fri, Apr 27, 2018 at 4:00 PM, David Lang <[email protected]> wrote: > On Fri, 27 Apr 2018, Flo Rance wrote: > > Ok cool. But I should admit that an example of a ruleset would help me a >> lot, because it's not easy to find some (good) documentation on how to >> build it. >> > > I agree we need to get some sample rulesets up. > > did you see the documentation at http://www.liblognorm.com/file > s/manual/index.html ? > > > David Lang > > On Fri, Apr 27, 2018 at 3:49 PM, David Lang <[email protected]> wrote: >> >> On Fri, 27 Apr 2018, Flo Rance wrote: >>> >>> I have taken a look at mmnormaliize and liblognorm and I've therefore >>> >>>> another question. >>>> >>>> Docker sends logs from different softwares to syslog, all of them with >>>> their own logging format. >>>> >>>> Will it be possible to create a ruleset to be applied for each programm >>>> and >>>> each format, in order to extract the severity ? And is it possible to >>>> normalize after having written the original log in a file ? >>>> >>>> >>> Yes, you would create a ruleset that has ruls for each program and it's >>> format. This can be done to any log message, it doesn't matter how >>> rsyslog >>> gathers the log >>> >>> David Lang >>> >>> >>> On Fri, Apr 27, 2018 at 9:26 AM, Flo Rance <[email protected]> wrote: >>> >>>> >>>> Ok, thank you guys, it's much more obvious. >>>> >>>>> >>>>> As a side note, CEST is not the programname, but the timezone (Central >>>>> European Summer Time). So it seems that the programname is missing and >>>>> only the pid is displayed. >>>>> >>>>> Anyway, I would really like to know how to use this mmnormalize module >>>>> to >>>>> parse the message and thus being able to forwared only relevant >>>>> messages. >>>>> >>>>> On Thu, Apr 26, 2018 at 8:22 PM, David Lang <[email protected]> wrote: >>>>> >>>>> On Thu, 26 Apr 2018, Rainer Gerhards wrote: >>>>> >>>>>> >>>>>> 2018-04-26 10:28 GMT+02:00 Flo Rance <[email protected]>: >>>>>> >>>>>> >>>>>>> Good idea. >>>>>>> >>>>>>>> >>>>>>>> Here's a typical message that is sent by docker to syslog: >>>>>>>> >>>>>>>> Debug line with all properties: >>>>>>>> FROMHOST: 'sc006692.domain', fromhost-ip: '127.0.0.1', HOSTNAME: >>>>>>>> 'sc006692.domain', PRI: 155, >>>>>>>> >>>>>>>> >>>>>>>> OK, rsyslog is doing the right thing. The PRI inside the message >>>>>>> (see >>>>>>> rawmsg below) is 155. Accoding to RFC5424, Sect. 6.2.1 that is a >>>>>>> facility of 19 and a severity of 3, or according to tables one and >>>>>>> two >>>>>>> in that section local3 with error severity. >>>>>>> >>>>>>> >>>>>>> so the docker log-driver is setting the severity to error >>>>>> >>>>>> syslogtag 'docker_fluance-ehealthdb[1116]:', programname: >>>>>> >>>>>> 'docker_fluance-ehealthdb', APP-NAME: 'docker_fluance-ehealthdb', >>>>>>> >>>>>>>> PROCID: >>>>>>>> '1116', MSGID: '-', >>>>>>>> TIMESTAMP: 'Apr 26 10:22:45', STRUCTURED-DATA: '-', >>>>>>>> msg: ' 2018-04-26 10:22:45.283 CEST [13518] postgres@postgres >>>>>>>> [local] >>>>>>>> FATAL: >>>>>>>> no pg_hba.conf entry for host "[local]", user "postgres", database >>>>>>>> "postgres", SSL off' >>>>>>>> >>>>>>>> >>>>>>>> But in the message itself, it is providing different information, >>>>>>> >>>>>> including a timestamp (2018-04-26 10:22:45.283) programname (CEST) pid >>>>>> (13518) and finally severity (FATAL:) >>>>>> >>>>>> escaped msg: ' 2018-04-26 10:22:45.283 CEST [13518] postgres@postgres >>>>>> >>>>>> [local] FATAL: no pg_hba.conf entry for host "[local]", user >>>>>>> >>>>>>>> "postgres", >>>>>>>> database "postgres", SSL off' >>>>>>>> inputname: imuxsock >>>>>>>> >>>>>>>> >>>>>>>> rawmsg: '<155>Apr 26 10:22:45 >>>>>>> >>>>>> >>>>>> docker_fluance-ehealthdb[1116]: 2018-04-26 10:22:45.283 CEST [13518] >>>>>>> >>>>>>>> postgres@postgres [local] FATAL: no pg_hba.conf entry for host >>>>>>>> "[local]", >>>>>>>> user "postgres", database "postgres", SSL off' >>>>>>>> >>>>>>>> >>>>>>>> If you look at the rawmsg, which is what the docker log-driver >>>>>>> passes >>>>>>> >>>>>> to >>>>>> rsyslog, you will see that it sets the pri to 155, the timestamp to >>>>>> Apr >>>>>> 26 >>>>>> 10:22:45 and the programname/pid to docker_fluance-ehealthdb[1116] >>>>>> >>>>>> the docker log driver makes the _assumption_ that what is spit out to >>>>>> stdout is not structured, so it adds a default header and passes it to >>>>>> rsyslog. >>>>>> >>>>>> I have to run to a meeting, I'll post some info later for configuring >>>>>> mmnormalize to parse the message. >>>>>> >>>>>> David Lang >>>>>> >>>>>> >>>>>> $!: >>>>>> >>>>>> $.: >>>>>>> >>>>>>>> $/: >>>>>>>> >>>>>>>> On Thu, Apr 26, 2018 at 9:57 AM, Rainer Gerhards < >>>>>>>> [email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> 2018-04-26 9:48 GMT+02:00 Flo Rance via rsyslog >>>>>>>>> <[email protected]>: >>>>>>>>> >>>>>>>>> What do you mean by "not in any standard format" ? >>>>>>>>> >>>>>>>>>> I've explicitely declared the logging driver in docker to be >>>>>>>>>> syslog, >>>>>>>>>> so >>>>>>>>>> I >>>>>>>>>> was expecting to be able to parse the result to extract accurate >>>>>>>>>> data >>>>>>>>>> in >>>>>>>>>> the fields. >>>>>>>>>> >>>>>>>>>> Can you give me any hints on how I could use mmnormalize to >>>>>>>>>> extract >>>>>>>>>> the >>>>>>>>>> various fields ? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> let's get started with something easier. Please add >>>>>>>>> >>>>>>>>> *.* /var/log/messagedebug;RSYSLOG_DebugFormat >>>>>>>>> >>>>>>>>> to the top of your rsyslog.conf. Let some messages flow. In the new >>>>>>>>> file, you should then see that message together with the decoded >>>>>>>>> properties AND the raw message. Post that entry (~6 lines or so). >>>>>>>>> With >>>>>>>>> that, we know for sure what is going on. >>>>>>>>> >>>>>>>>> Rainer >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Apr 25, 2018 at 7:28 PM, David Lang <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>> On Wed, 25 Apr 2018, Rainer Gerhards wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 2018-04-25 9:29 GMT+02:00 Flo Rance <[email protected]>: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Ok, but if ".err" means "err and above", why does it forward >>>>>>>>>>>> >>>>>>>>>>>> messages >>>>>>>>>>>>> with >>>>>>>>>>>>> the severity INFO as in the example ? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> pls post the raw message - how do you know it is INFO? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> in the docker world, the 'standard' is that messages get dumped >>>>>>>>>>>> to >>>>>>>>>>>> >>>>>>>>>>> stdout, >>>>>>>>>>> not in any standard format, so INFO: in the message body is the >>>>>>>>>>> indication. >>>>>>>>>>> >>>>>>>>>>> It looks like these logs should be parsed with mmnormalize to >>>>>>>>>>> extract >>>>>>>>>>> the >>>>>>>>>>> various fields (potentially as a parser on the input) >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> >>>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>>> myriad >>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>>> you DON'T >>>>>>>>>> LIKE THAT. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
