Hi David and Peter, The applications we're deploying are running inside docker containers. They have no direct access to a syslog socket. While it's possible to bind-mount a syslog socket into a container, the considered best practice is to log direct to stdout. We log JSON because we have some requirements for capturing metadata and reporting machine-readable metrics.
Until recently we were capturing that stdout into files on disk, and then using logstash-forwarder to ship it to Elasticsearch. We're now rebuilding our logging stack, and are planning on using rsyslogd to ship and transform logs. Usually we only ship INFO level and above to the central logging server so that we don't drown in noise, so we would like to store DEBUG logs in the local journal for troubleshooting in case of crisis. The proposed solution looks like this: Docker Container logs to stdout -> Docker Daemon captures stdout and directs to syslog -> Rsyslog places human readable text into the journal and ships the raw json to redis for later processing. The problem is that the docker daemon is only capturing *raw text on stdout*, and is not able to discern the appropriate log level. Anything logged to stdout is tagged with LOG_INFO, anything on stderr ships with LOG_ERR. I have free reign over the message content, but no useful control of the headers. I've considered trying to extend the syslog driver in Docker, but realistically that means implementing the kind of introspection and transformation at which rsyslog already excels. Yours, -- B On 3 February 2016 at 20:24, David Lang <[email protected]> wrote: > On Wed, 3 Feb 2016, Bob Gregory wrote: > > Hi all, >> >> I'm using rsyslogd as the syslog daemon on a machine running Docker. I've >> configured docker to use the syslog logging driver and am able to parse >> the >> json logs written to stdout by my applications. >> >> These logs contain a textual severity level based on the log4j levels: >> DEBUG, INFO, WARN, ERROR, CRITICAL, FATAL. >> >> The docker syslog integration dumps all the stdout of a container into >> syslog with a severity of LOG_INFO, and stderr with LOG_ERR. >> >> I'd like to parse the incoming json and map the level names to syslog >> severity numbers. >> >> I can see some related functionality in msg.c, but nothing that's exposed >> to end users, so I'm considering writing a new pair of property replacers: >> one to map numbers from standard error level or severity names; another to >> map severity levels to their names: >> >> template(name="my-magic-template") { >> property(name="$!level" severity.fromname="1") >> property(name="$!levelno" severity.toname="1") >> } >> >> template(name="my-other-template" string="%level::severity-from-name% >> %levelno::severity-to-name%") >> >> Has anyone got any better ideas? I'd like to continue logging from >> containers to stdout, and to continue using the log-level names, because >> the php/python/java logging libs support that out-of-the-box and it's one >> less thing for devs to worry about. >> > > a properly formatted log message is going to contain the facility/severity > information in the header of the message as a numberic value that rsyslog > parses for you. > > is there a way to get this from the docker stuff as syslog messages rather > than just raw json? Ideally you get JSON as the body of the syslog message, > so you have the header formatted properly and then have the message details > in JSON for easy parsing > > log a few messages with the template RSYSLOG_DebugFormat you may find that > this is done properly and you don't have to fight it. > > David Lang > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- ---- *Bob Gregory* Application Architect MADE.COM <http://www.made.com/> Skype: flinkywistypomm [image: MADE] Made.com Design Limited is a company registered in England and Wales. Registered number: 07101408 | Registered office: 100 Charing Cross Road, London WC2H 0HG _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
