Hi Dave, It's the latter. Currently docker is just spraying logs out onto disk, in both plain text and json format, and there's no logrotate. Instead, we want just the json logs to go through rsyslog. We'll forward INFO level application logs to Elasticsearch via Redis, and put a human-readable version of logs into the journal.
Marking the journal entries with the appropriate syslog severity makes it easy to query and filter. The lookup_table functionality actually works better than my proposed property replacer, because it's simple to modify the lookup if requirements evolve. -- B On 4 February 2016 at 15:15, Dave Caplinger <[email protected]> wrote: > Hi Bob, > > I'm curious to better understand your objective: > > > [some docker container] logs contain a textual severity level based on > the log4j levels: > > DEBUG, INFO, WARN, ERROR, CRITICAL, FATAL. > > > > The docker syslog integration dumps all the stdout of a container into > > syslog with a severity of LOG_INFO, and stderr with LOG_ERR. > > > > I'd like to parse the incoming json and map the level names to syslog > > severity numbers. > > Is it correct that you're trying to maintain the distinction between the > LOG_INFO and LOG_ERR streams coming out of the docker containers? If > that's *all* you're trying to achieve, would just adding another property > to your JSON output to store the info/err value be sufficient? > > Or... do you have existing downstream log processing that depends on the > syslog severity values, so mapping the log4j {DEBUG, INFO, WARN, ERROR, > CRITICAL, FATAL} text values onto syslog {Debug, Info, Notice, Warning, > Error, Critical, Alert, Emergency} severities is the critical part? > > (Or some other scenario I haven't understood yet ...?) > > -- > Dave Caplinger | Director, Technical Product Management > Solutionary — An NTT Group Security Company > > > On Feb 4, 2016, at 7:16 AM, Bob Gregory <[email protected]> wrote: > > > > Hi David, > > > > We ran logstash-forwarder in a separate container, and shared volumes > > between app containers and a forwarding container. That's problematic as > we > > move toward a clustered environment, because it means running multiple > > instances of logstash forwarder, or doing something peculiar with user > > permissions. Instead we'd like to delegate the log routing and filtering > to > > the host OS via Docker's log driver. > > > > > > On 4 February 2016 at 11:40, David Lang <[email protected]> wrote: > > > >> On Thu, 4 Feb 2016, Bob Gregory wrote: > >> > >> can you syslog over the network to localhost? rsyslog can be pretty > >>>> > >>> lightweight if you set queue sizes smaller than the default 500K > messages. > >>> If you were running logstash-forwarder, rsyslog should be lighter. > >>> > >>> That's an interesting idea, but it would mean running a syslog daemon > in > >>> each container. Generally, we stick to a single foreground process per > >>> container, so there's no init system for managing daemonised services, > but > >>> that might change in the future. > >>> > >> > >> how were you running the logstash forwarder then? > >> > >> > >> David Lang > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > > > > > > > -- > > > > ---- > > > > *Bob Gregory* > > > > Application Architect > > > > MADE.COM <http://www.made.com/> > > > > Skype: flinkywistypomm > > > > > > [image: MADE] > > > > > > > > Made.com Design Limited is a company registered in England and Wales. > > > > Registered number: 07101408 | Registered office: 100 Charing Cross Road, > > London WC2H 0HG > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- ---- *Bob Gregory* Application Architect MADE.COM <http://www.made.com/> Skype: flinkywistypomm [image: MADE] Made.com Design Limited is a company registered in England and Wales. Registered number: 07101408 | Registered office: 100 Charing Cross Road, London WC2H 0HG _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
