To be clear, there are several simple solutions - I could just require all applications to log a syslog severity number in their json output - but a) that would be no fun, b) that might mean writing mapping code in each app, and c) I thought this might be a problem others had encountered for which I could proffer a simple fix.
Yours playfully, -- Bob On 3 February 2016 at 20:57, Bob Gregory <[email protected]> wrote: > Hi David and Peter, > > The applications we're deploying are running inside docker containers. > They have no direct access to a syslog socket. While it's possible to > bind-mount a syslog socket into a container, the considered best practice > is to log direct to stdout. We log JSON because we have some requirements > for capturing metadata and reporting machine-readable metrics. > > Until recently we were capturing that stdout into files on disk, and then > using logstash-forwarder to ship it to Elasticsearch. We're now rebuilding > our logging stack, and are planning on using rsyslogd to ship and transform > logs. > > Usually we only ship INFO level and above to the central logging server so > that we don't drown in noise, so we would like to store DEBUG logs in the > local journal for troubleshooting in case of crisis. > > The proposed solution looks like this: > > Docker Container logs to stdout -> Docker Daemon captures stdout and > directs to syslog -> Rsyslog places human readable text into the journal > and ships the raw json to redis for later processing. > > > The problem is that the docker daemon is only capturing *raw text on > stdout*, and is not able to discern the appropriate log level. Anything > logged to stdout is tagged with LOG_INFO, anything on stderr ships with > LOG_ERR. I have free reign over the message content, but no useful control > of the headers. I've considered trying to extend the syslog driver in > Docker, but realistically that means implementing the kind of introspection > and transformation at which rsyslog already excels. > > > Yours, > > -- B > > On 3 February 2016 at 20:24, David Lang <[email protected]> wrote: > >> On Wed, 3 Feb 2016, Bob Gregory wrote: >> >> Hi all, >>> >>> I'm using rsyslogd as the syslog daemon on a machine running Docker. I've >>> configured docker to use the syslog logging driver and am able to parse >>> the >>> json logs written to stdout by my applications. >>> >>> These logs contain a textual severity level based on the log4j levels: >>> DEBUG, INFO, WARN, ERROR, CRITICAL, FATAL. >>> >>> The docker syslog integration dumps all the stdout of a container into >>> syslog with a severity of LOG_INFO, and stderr with LOG_ERR. >>> >>> I'd like to parse the incoming json and map the level names to syslog >>> severity numbers. >>> >>> I can see some related functionality in msg.c, but nothing that's exposed >>> to end users, so I'm considering writing a new pair of property >>> replacers: >>> one to map numbers from standard error level or severity names; another >>> to >>> map severity levels to their names: >>> >>> template(name="my-magic-template") { >>> property(name="$!level" severity.fromname="1") >>> property(name="$!levelno" severity.toname="1") >>> } >>> >>> template(name="my-other-template" string="%level::severity-from-name% >>> %levelno::severity-to-name%") >>> >>> Has anyone got any better ideas? I'd like to continue logging from >>> containers to stdout, and to continue using the log-level names, because >>> the php/python/java logging libs support that out-of-the-box and it's one >>> less thing for devs to worry about. >>> >> >> a properly formatted log message is going to contain the >> facility/severity information in the header of the message as a numberic >> value that rsyslog parses for you. >> >> is there a way to get this from the docker stuff as syslog messages >> rather than just raw json? Ideally you get JSON as the body of the syslog >> message, so you have the header formatted properly and then have the >> message details in JSON for easy parsing >> >> log a few messages with the template RSYSLOG_DebugFormat you may find >> that this is done properly and you don't have to fight it. >> >> David Lang >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > > > -- > > ---- > > *Bob Gregory* > > Application Architect > > MADE.COM <http://www.made.com/> > > Skype: flinkywistypomm > > > [image: MADE] > > > > Made.com Design Limited is a company registered in England and Wales. > > Registered number: 07101408 | Registered office: 100 Charing Cross Road, > London WC2H 0HG > -- ---- *Bob Gregory* Application Architect MADE.COM <http://www.made.com/> Skype: flinkywistypomm [image: MADE] Made.com Design Limited is a company registered in England and Wales. Registered number: 07101408 | Registered office: 100 Charing Cross Road, London WC2H 0HG _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
