On Fri, 27 Apr 2018, Flo Rance wrote:
Yes, I did and I was somehow lost in the section "Rules".
http://www.liblognorm.com/files/manual/configuration.html#rulebase
Concretely, my idea is to normalize the msg part:
msg: ' 2018-04-26 10:22:45.283 CEST [13518] postgres@postgres [local]
FATAL: no pg_hba.conf entry for host "[local]", user "postgres", database
"postgres", SSL off'
so this would start off with something like
rule=: %date:date-iso% %time:time-24hr% %tz:word% [%pid:char-to:]%] %db:word%
%place:word% %severity:word% %msg:rest%
(I'm not sure if time-24 hour will handle the decimal seconds, and you may be
able to use date-rfc5424 for this, and it may or may not include the timezone)
try parsing the log with this rule and look at the contents of the resulting
variables (lognormalizer will be the frist step in this, see
http://www.liblognorm.com/files/manual/lognormalizer.html then you can put this
into rsyslog and log with the format RSYSLOG_DebugFormat to see how the
variables created by the rule show up in rsyslog.
note, this doesn't affect $severity, it creates a new variable $!severity that
you would then have to make use of, in a rule or a template.
David Lang
However, not so much examples treat about severity.
On Fri, Apr 27, 2018 at 4:00 PM, David Lang <[email protected]> wrote:
On Fri, 27 Apr 2018, Flo Rance wrote:
Ok cool. But I should admit that an example of a ruleset would help me a
lot, because it's not easy to find some (good) documentation on how to
build it.
I agree we need to get some sample rulesets up.
did you see the documentation at http://www.liblognorm.com/file
s/manual/index.html ?
David Lang
On Fri, Apr 27, 2018 at 3:49 PM, David Lang <[email protected]> wrote:
On Fri, 27 Apr 2018, Flo Rance wrote:
I have taken a look at mmnormaliize and liblognorm and I've therefore
another question.
Docker sends logs from different softwares to syslog, all of them with
their own logging format.
Will it be possible to create a ruleset to be applied for each programm
and
each format, in order to extract the severity ? And is it possible to
normalize after having written the original log in a file ?
Yes, you would create a ruleset that has ruls for each program and it's
format. This can be done to any log message, it doesn't matter how
rsyslog
gathers the log
David Lang
On Fri, Apr 27, 2018 at 9:26 AM, Flo Rance <[email protected]> wrote:
Ok, thank you guys, it's much more obvious.
As a side note, CEST is not the programname, but the timezone (Central
European Summer Time). So it seems that the programname is missing and
only the pid is displayed.
Anyway, I would really like to know how to use this mmnormalize module
to
parse the message and thus being able to forwared only relevant
messages.
On Thu, Apr 26, 2018 at 8:22 PM, David Lang <[email protected]> wrote:
On Thu, 26 Apr 2018, Rainer Gerhards wrote:
2018-04-26 10:28 GMT+02:00 Flo Rance <[email protected]>:
Good idea.
Here's a typical message that is sent by docker to syslog:
Debug line with all properties:
FROMHOST: 'sc006692.domain', fromhost-ip: '127.0.0.1', HOSTNAME:
'sc006692.domain', PRI: 155,
OK, rsyslog is doing the right thing. The PRI inside the message
(see
rawmsg below) is 155. Accoding to RFC5424, Sect. 6.2.1 that is a
facility of 19 and a severity of 3, or according to tables one and
two
in that section local3 with error severity.
so the docker log-driver is setting the severity to error
syslogtag 'docker_fluance-ehealthdb[1116]:', programname:
'docker_fluance-ehealthdb', APP-NAME: 'docker_fluance-ehealthdb',
PROCID:
'1116', MSGID: '-',
TIMESTAMP: 'Apr 26 10:22:45', STRUCTURED-DATA: '-',
msg: ' 2018-04-26 10:22:45.283 CEST [13518] postgres@postgres
[local]
FATAL:
no pg_hba.conf entry for host "[local]", user "postgres", database
"postgres", SSL off'
But in the message itself, it is providing different information,
including a timestamp (2018-04-26 10:22:45.283) programname (CEST) pid
(13518) and finally severity (FATAL:)
escaped msg: ' 2018-04-26 10:22:45.283 CEST [13518] postgres@postgres
[local] FATAL: no pg_hba.conf entry for host "[local]", user
"postgres",
database "postgres", SSL off'
inputname: imuxsock
rawmsg: '<155>Apr 26 10:22:45
docker_fluance-ehealthdb[1116]: 2018-04-26 10:22:45.283 CEST [13518]
postgres@postgres [local] FATAL: no pg_hba.conf entry for host
"[local]",
user "postgres", database "postgres", SSL off'
If you look at the rawmsg, which is what the docker log-driver
passes
to
rsyslog, you will see that it sets the pri to 155, the timestamp to
Apr
26
10:22:45 and the programname/pid to docker_fluance-ehealthdb[1116]
the docker log driver makes the _assumption_ that what is spit out to
stdout is not structured, so it adds a default header and passes it to
rsyslog.
I have to run to a meeting, I'll post some info later for configuring
mmnormalize to parse the message.
David Lang
$!:
$.:
$/:
On Thu, Apr 26, 2018 at 9:57 AM, Rainer Gerhards <
[email protected]>
wrote:
2018-04-26 9:48 GMT+02:00 Flo Rance via rsyslog
<[email protected]>:
What do you mean by "not in any standard format" ?
I've explicitely declared the logging driver in docker to be
syslog,
so
I
was expecting to be able to parse the result to extract accurate
data
in
the fields.
Can you give me any hints on how I could use mmnormalize to
extract
the
various fields ?
let's get started with something easier. Please add
*.* /var/log/messagedebug;RSYSLOG_DebugFormat
to the top of your rsyslog.conf. Let some messages flow. In the new
file, you should then see that message together with the decoded
properties AND the raw message. Post that entry (~6 lines or so).
With
that, we know for sure what is going on.
Rainer
On Wed, Apr 25, 2018 at 7:28 PM, David Lang <[email protected]> wrote:
On Wed, 25 Apr 2018, Rainer Gerhards wrote:
2018-04-25 9:29 GMT+02:00 Flo Rance <[email protected]>:
Ok, but if ".err" means "err and above", why does it forward
messages
with
the severity INFO as in the example ?
pls post the raw message - how do you know it is INFO?
in the docker world, the 'standard' is that messages get dumped
to
stdout,
not in any standard format, so INFO: in the message body is the
indication.
It looks like these logs should be parsed with mmnormalize to
extract
the
various fields (potentially as a parser on the input)
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
