the table_lookup function that's being updated for 8.17 would let you do
arbitrary lookups including this.
but looking at your problem
if you are just catching stdout and configuring the app to write in JSON, then
you can't depend on the messages having anything in them (and can't get messages
from unmodified software/libraries that are trying to log to /dev/log)
can you syslog over the network to localhost? rsyslog can be pretty lightweight
if you set queue sizes smaller than the default 500K messages. If you were
running logstash-forwarder, rsyslog should be lighter.
If your docker containers aren't dynamic, rsyslog can open a log socket in each
one (restart rsyslog with the appropriate config after the containers are
created). I've got an enhancement request in to let us add/remove such sockets
on the fly
you really are better off using the syslog interface (with a JSON structured
message body) than writing to stdout and intercepting.
David Lang
On Wed, 3 Feb 2016, Bob Gregory wrote:
Date: Wed, 3 Feb 2016 21:05:50 +0000
From: Bob Gregory <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] Mapping log-level names to severity numbers
To be clear, there are several simple solutions - I could just require all
applications to log a syslog severity number in their json output - but
a) that would be no fun,
b) that might mean writing mapping code in each app,
and c) I thought this might be a problem others had encountered for which
I could proffer a simple fix.
Yours playfully,
-- Bob
On 3 February 2016 at 20:57, Bob Gregory <[email protected]> wrote:
Hi David and Peter,
The applications we're deploying are running inside docker containers.
They have no direct access to a syslog socket. While it's possible to
bind-mount a syslog socket into a container, the considered best practice
is to log direct to stdout. We log JSON because we have some requirements
for capturing metadata and reporting machine-readable metrics.
Until recently we were capturing that stdout into files on disk, and then
using logstash-forwarder to ship it to Elasticsearch. We're now rebuilding
our logging stack, and are planning on using rsyslogd to ship and transform
logs.
Usually we only ship INFO level and above to the central logging server so
that we don't drown in noise, so we would like to store DEBUG logs in the
local journal for troubleshooting in case of crisis.
The proposed solution looks like this:
Docker Container logs to stdout -> Docker Daemon captures stdout and
directs to syslog -> Rsyslog places human readable text into the journal
and ships the raw json to redis for later processing.
The problem is that the docker daemon is only capturing *raw text on
stdout*, and is not able to discern the appropriate log level. Anything
logged to stdout is tagged with LOG_INFO, anything on stderr ships with
LOG_ERR. I have free reign over the message content, but no useful control
of the headers. I've considered trying to extend the syslog driver in
Docker, but realistically that means implementing the kind of introspection
and transformation at which rsyslog already excels.
Yours,
-- B
On 3 February 2016 at 20:24, David Lang <[email protected]> wrote:
On Wed, 3 Feb 2016, Bob Gregory wrote:
Hi all,
I'm using rsyslogd as the syslog daemon on a machine running Docker. I've
configured docker to use the syslog logging driver and am able to parse
the
json logs written to stdout by my applications.
These logs contain a textual severity level based on the log4j levels:
DEBUG, INFO, WARN, ERROR, CRITICAL, FATAL.
The docker syslog integration dumps all the stdout of a container into
syslog with a severity of LOG_INFO, and stderr with LOG_ERR.
I'd like to parse the incoming json and map the level names to syslog
severity numbers.
I can see some related functionality in msg.c, but nothing that's exposed
to end users, so I'm considering writing a new pair of property
replacers:
one to map numbers from standard error level or severity names; another
to
map severity levels to their names:
template(name="my-magic-template") {
property(name="$!level" severity.fromname="1")
property(name="$!levelno" severity.toname="1")
}
template(name="my-other-template" string="%level::severity-from-name%
%levelno::severity-to-name%")
Has anyone got any better ideas? I'd like to continue logging from
containers to stdout, and to continue using the log-level names, because
the php/python/java logging libs support that out-of-the-box and it's one
less thing for devs to worry about.
a properly formatted log message is going to contain the
facility/severity information in the header of the message as a numberic
value that rsyslog parses for you.
is there a way to get this from the docker stuff as syslog messages
rather than just raw json? Ideally you get JSON as the body of the syslog
message, so you have the header formatted properly and then have the
message details in JSON for easy parsing
log a few messages with the template RSYSLOG_DebugFormat you may find
that this is done properly and you don't have to fight it.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
--
----
*Bob Gregory*
Application Architect
MADE.COM <http://www.made.com/>
Skype: flinkywistypomm
[image: MADE]
Made.com Design Limited is a company registered in England and Wales.
Registered number: 07101408 | Registered office: 100 Charing Cross Road,
London WC2H 0HG
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
