On 15 April 2015 at 20:16, David Lang <[email protected]> wrote: > This is why I love opensource software, comment on a need and someone will > point at a tool. > > Thanks! > wq > > On Wed, 15 Apr 2015, Brian Knox wrote: > > We keep our logs in JSON format and don't find it to be a drawback. We >> have logs searchable in elasticsearch - and for working with logs on disk, >> have a small program that logs can be piped through that strips out >> everything but the json which makes it very easy to pipe logs to jq (a >> command line json processor - see https://stedolan.github.io/jq/ ). >> >> We've done something similar - we get our applications to output RFC 5424 directly to rsyslog and then ship the logs to a central store and to Elastic Search. The application's logging library must support RFC 5424 though. This has that processing is slightly more distributed and the number of transformations is reduced slightly.
> >> >> On Wed, Apr 15, 2015 at 12:48 AM, David Lang <[email protected]> wrote: >> >> On Wed, 15 Apr 2015, Ezell, Matthew A. wrote: >>> >>> Sure, as a system administrator it's pretty clear how best to handle >>> this. >>> >>>> If there's CEE JSON data coming over the wire, use mmjsonparse. If it's >>>> unstructured traditional syslog(3) data, use mmnormalize to try to >>>> extract >>>> relevant fields based on rules I setup. Write the traditional "message" >>>> field to /var/log/messages and send the structured data to >>>> ElasticSearch. >>>> But I'm a system administrator who cares about structured logging, so I >>>> would have a custom rsyslog setup to handle this seamlessly. >>>> >>>> The question is really from the application developer's point of view. >>>> How do you log structured data in a way that doesn't change the format >>>> of >>>> /var/log/messages for most users, but provides additional information >>>> for >>>> those system administrators who choose to handle the structured data? >>>> >>>> Imagine going to the developers of OpenSSH and requesting that they >>>> start >>>> logging structured data. If they simply changed all their syslog(3) >>>> calls >>>> to output CEE JSON instead instead of plain strings, it's going to break >>>> just about every brute-force login detection system out there. That's >>>> unacceptable. What is the *right* thing for them to do? >>>> >>>> >>> do like ossec does and have a config option that switches to JSON output. >>> >>> since they have to have their software work everywhere that it's working >>> today, they can't change it's output at all. anything they do will break >>> parsers. >>> >>> but with a config switch (which a distro could turn on by default), they >>> can output a different format, and that format could be JSON with the old >>> log text in a msg field (again though, which is the source of truth if >>> they >>> differ) >>> >>> David Lang >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- Kind regards, Brett Delle Grazie _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
