On Fri, 27 Apr 2018, Flo Rance wrote:
I have taken a look at mmnormaliize and liblognorm and I've therefore
another question.
Docker sends logs from different softwares to syslog, all of them with
their own logging format.
Will it be possible to create a ruleset to be applied for each programm and
each format, in order to extract the severity ? And is it possible to
normalize after having written the original log in a file ?
Yes, you would create a ruleset that has ruls for each program and it's format.
This can be done to any log message, it doesn't matter how rsyslog gathers the
log
David Lang
On Fri, Apr 27, 2018 at 9:26 AM, Flo Rance <[email protected]> wrote:
Ok, thank you guys, it's much more obvious.
As a side note, CEST is not the programname, but the timezone (Central
European Summer Time). So it seems that the programname is missing and
only the pid is displayed.
Anyway, I would really like to know how to use this mmnormalize module to
parse the message and thus being able to forwared only relevant messages.
On Thu, Apr 26, 2018 at 8:22 PM, David Lang <[email protected]> wrote:
On Thu, 26 Apr 2018, Rainer Gerhards wrote:
2018-04-26 10:28 GMT+02:00 Flo Rance <[email protected]>:
Good idea.
Here's a typical message that is sent by docker to syslog:
Debug line with all properties:
FROMHOST: 'sc006692.domain', fromhost-ip: '127.0.0.1', HOSTNAME:
'sc006692.domain', PRI: 155,
OK, rsyslog is doing the right thing. The PRI inside the message (see
rawmsg below) is 155. Accoding to RFC5424, Sect. 6.2.1 that is a
facility of 19 and a severity of 3, or according to tables one and two
in that section local3 with error severity.
so the docker log-driver is setting the severity to error
syslogtag 'docker_fluance-ehealthdb[1116]:', programname:
'docker_fluance-ehealthdb', APP-NAME: 'docker_fluance-ehealthdb',
PROCID:
'1116', MSGID: '-',
TIMESTAMP: 'Apr 26 10:22:45', STRUCTURED-DATA: '-',
msg: ' 2018-04-26 10:22:45.283 CEST [13518] postgres@postgres [local]
FATAL:
no pg_hba.conf entry for host "[local]", user "postgres", database
"postgres", SSL off'
But in the message itself, it is providing different information,
including a timestamp (2018-04-26 10:22:45.283) programname (CEST) pid
(13518) and finally severity (FATAL:)
escaped msg: ' 2018-04-26 10:22:45.283 CEST [13518] postgres@postgres
[local] FATAL: no pg_hba.conf entry for host "[local]", user
"postgres",
database "postgres", SSL off'
inputname: imuxsock
rawmsg: '<155>Apr 26 10:22:45
docker_fluance-ehealthdb[1116]: 2018-04-26 10:22:45.283 CEST [13518]
postgres@postgres [local] FATAL: no pg_hba.conf entry for host
"[local]",
user "postgres", database "postgres", SSL off'
If you look at the rawmsg, which is what the docker log-driver passes to
rsyslog, you will see that it sets the pri to 155, the timestamp to Apr 26
10:22:45 and the programname/pid to docker_fluance-ehealthdb[1116]
the docker log driver makes the _assumption_ that what is spit out to
stdout is not structured, so it adds a default header and passes it to
rsyslog.
I have to run to a meeting, I'll post some info later for configuring
mmnormalize to parse the message.
David Lang
$!:
$.:
$/:
On Thu, Apr 26, 2018 at 9:57 AM, Rainer Gerhards <
[email protected]>
wrote:
2018-04-26 9:48 GMT+02:00 Flo Rance via rsyslog
<[email protected]>:
What do you mean by "not in any standard format" ?
I've explicitely declared the logging driver in docker to be syslog,
so
I
was expecting to be able to parse the result to extract accurate data
in
the fields.
Can you give me any hints on how I could use mmnormalize to extract
the
various fields ?
let's get started with something easier. Please add
*.* /var/log/messagedebug;RSYSLOG_DebugFormat
to the top of your rsyslog.conf. Let some messages flow. In the new
file, you should then see that message together with the decoded
properties AND the raw message. Post that entry (~6 lines or so). With
that, we know for sure what is going on.
Rainer
On Wed, Apr 25, 2018 at 7:28 PM, David Lang <[email protected]> wrote:
On Wed, 25 Apr 2018, Rainer Gerhards wrote:
2018-04-25 9:29 GMT+02:00 Flo Rance <[email protected]>:
Ok, but if ".err" means "err and above", why does it forward
messages
with
the severity INFO as in the example ?
pls post the raw message - how do you know it is INFO?
in the docker world, the 'standard' is that messages get dumped to
stdout,
not in any standard format, so INFO: in the message body is the
indication.
It looks like these logs should be parsed with mmnormalize to extract
the
various fields (potentially as a parser on the input)
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
