Hi David, Excellent points all. I'll take a look at the table lookup, and reconsider whether we mount a syslog socket into containers. Just for clarity:
> if you are just catching stdout and configuring the app to write in JSON, then you can't depend on the messages having anything in them (and can't get messages from unmodified software/libraries that are trying to log to /dev/log) In practice, that's not an issue for two reasons: firstly, when running an application in a container, it's standard practice to run it in the foreground and send output to stdout. The only app we run that doesn't support that pattern is postfix. Secondly, I'm generally only interested in capturing *our* application logs as json. Things like nginx, HAProxy, or OpenVpn continue to log in plain text for troubleshooting but we only ship aggregate metrics to a remote server. It's simple to add a syslog severity as a <1-7> message prefix, but that'll clash with the @cee cookie we use for json messages. > can you syslog over the network to localhost? rsyslog can be pretty lightweight if you set queue sizes smaller than the default 500K messages. If you were running logstash-forwarder, rsyslog should be lighter. That's an interesting idea, but it would mean running a syslog daemon in each container. Generally, we stick to a single foreground process per container, so there's no init system for managing daemonised services, but that might change in the future. > If your docker containers aren't dynamic, rsyslog can open a log socket in each one (restart rsyslog with the appropriate config after the containers are created). I've got an enhancement request in to let us add/remove such sockets on the fly The docker containers are *very much* dynamic, but that mightn't be a problem. > you really are better off using the syslog interface (with a JSON structured message body) than writing to stdout and intercepting. Understood and largely agreed - I'm trying to find a reasonable balance between standard Docker practice, sympathetic use of rsyslog, and easy plug-and-play for developers. Thanks for your comments. -- B On 4 February 2016 at 02:16, David Lang <[email protected]> wrote: > the table_lookup function that's being updated for 8.17 would let you do > arbitrary lookups including this. > > but looking at your problem > > if you are just catching stdout and configuring the app to write in JSON, > then you can't depend on the messages having anything in them (and can't > get messages from unmodified software/libraries that are trying to log to > /dev/log) > > can you syslog over the network to localhost? rsyslog can be pretty > lightweight if you set queue sizes smaller than the default 500K messages. > If you were running logstash-forwarder, rsyslog should be lighter. > > If your docker containers aren't dynamic, rsyslog can open a log socket in > each one (restart rsyslog with the appropriate config after the containers > are created). I've got an enhancement request in to let us add/remove such > sockets on the fly > > you really are better off using the syslog interface (with a JSON > structured message body) than writing to stdout and intercepting. > > David Lang > > > On Wed, 3 Feb 2016, Bob Gregory wrote: > > Date: Wed, 3 Feb 2016 21:05:50 +0000 >> From: Bob Gregory <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] Mapping log-level names to severity numbers >> >> >> To be clear, there are several simple solutions - I could just require all >> applications to log a syslog severity number in their json output - but >> a) that would be no fun, >> b) that might mean writing mapping code in each app, >> and c) I thought this might be a problem others had encountered for which >> I could proffer a simple fix. >> >> Yours playfully, >> >> -- Bob >> >> On 3 February 2016 at 20:57, Bob Gregory <[email protected]> wrote: >> >> Hi David and Peter, >>> >>> The applications we're deploying are running inside docker containers. >>> They have no direct access to a syslog socket. While it's possible to >>> bind-mount a syslog socket into a container, the considered best practice >>> is to log direct to stdout. We log JSON because we have some requirements >>> for capturing metadata and reporting machine-readable metrics. >>> >>> Until recently we were capturing that stdout into files on disk, and then >>> using logstash-forwarder to ship it to Elasticsearch. We're now >>> rebuilding >>> our logging stack, and are planning on using rsyslogd to ship and >>> transform >>> logs. >>> >>> Usually we only ship INFO level and above to the central logging server >>> so >>> that we don't drown in noise, so we would like to store DEBUG logs in the >>> local journal for troubleshooting in case of crisis. >>> >>> The proposed solution looks like this: >>> >>> Docker Container logs to stdout -> Docker Daemon captures stdout and >>> directs to syslog -> Rsyslog places human readable text into the journal >>> and ships the raw json to redis for later processing. >>> >>> >>> The problem is that the docker daemon is only capturing *raw text on >>> stdout*, and is not able to discern the appropriate log level. Anything >>> logged to stdout is tagged with LOG_INFO, anything on stderr ships with >>> LOG_ERR. I have free reign over the message content, but no useful >>> control >>> of the headers. I've considered trying to extend the syslog driver in >>> Docker, but realistically that means implementing the kind of >>> introspection >>> and transformation at which rsyslog already excels. >>> >>> >>> Yours, >>> >>> -- B >>> >>> On 3 February 2016 at 20:24, David Lang <[email protected]> wrote: >>> >>> On Wed, 3 Feb 2016, Bob Gregory wrote: >>>> >>>> Hi all, >>>> >>>>> >>>>> I'm using rsyslogd as the syslog daemon on a machine running Docker. >>>>> I've >>>>> configured docker to use the syslog logging driver and am able to parse >>>>> the >>>>> json logs written to stdout by my applications. >>>>> >>>>> These logs contain a textual severity level based on the log4j levels: >>>>> DEBUG, INFO, WARN, ERROR, CRITICAL, FATAL. >>>>> >>>>> The docker syslog integration dumps all the stdout of a container into >>>>> syslog with a severity of LOG_INFO, and stderr with LOG_ERR. >>>>> >>>>> I'd like to parse the incoming json and map the level names to syslog >>>>> severity numbers. >>>>> >>>>> I can see some related functionality in msg.c, but nothing that's >>>>> exposed >>>>> to end users, so I'm considering writing a new pair of property >>>>> replacers: >>>>> one to map numbers from standard error level or severity names; another >>>>> to >>>>> map severity levels to their names: >>>>> >>>>> template(name="my-magic-template") { >>>>> property(name="$!level" severity.fromname="1") >>>>> property(name="$!levelno" severity.toname="1") >>>>> } >>>>> >>>>> template(name="my-other-template" string="%level::severity-from-name% >>>>> %levelno::severity-to-name%") >>>>> >>>>> Has anyone got any better ideas? I'd like to continue logging from >>>>> containers to stdout, and to continue using the log-level names, >>>>> because >>>>> the php/python/java logging libs support that out-of-the-box and it's >>>>> one >>>>> less thing for devs to worry about. >>>>> >>>>> >>>> a properly formatted log message is going to contain the >>>> facility/severity information in the header of the message as a numberic >>>> value that rsyslog parses for you. >>>> >>>> is there a way to get this from the docker stuff as syslog messages >>>> rather than just raw json? Ideally you get JSON as the body of the >>>> syslog >>>> message, so you have the header formatted properly and then have the >>>> message details in JSON for easy parsing >>>> >>>> log a few messages with the template RSYSLOG_DebugFormat you may find >>>> that this is done properly and you don't have to fight it. >>>> >>>> David Lang >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> >>>> >>> >>> >>> -- >>> >>> ---- >>> >>> *Bob Gregory* >>> >>> Application Architect >>> >>> MADE.COM <http://www.made.com/> >>> >>> Skype: flinkywistypomm >>> >>> >>> [image: MADE] >>> >>> >>> >>> Made.com Design Limited is a company registered in England and Wales. >>> >>> Registered number: 07101408 | Registered office: 100 Charing Cross Road, >>> London WC2H 0HG >>> >>> >> >> >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- ---- *Bob Gregory* Application Architect MADE.COM <http://www.made.com/> Skype: flinkywistypomm [image: MADE] Made.com Design Limited is a company registered in England and Wales. Registered number: 07101408 | Registered office: 100 Charing Cross Road, London WC2H 0HG _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
