That's interesting - dropping the cee cookie gives my beleaguered devs one less thing to complain about, and I'll probably use mmnormalize anyway for processing things like nginx and haproxy logs into the journal.
I'll have a play tomorrow and see how it goes - thanks again David. On 4 February 2016 at 19:11, David Lang <[email protected]> wrote: > oops, libfastjson > > https://github.com/rsyslog/libfastjson > > http://blog.gerhards.net/2015/12/rsyslog-and-liblognorm-will-switch-to.html > > I think the hard requirement for libfastjson just hit in the last day or > so. liblognorm will still use json-c (but will use libfastjson if it finds > it), but as of last night, rsyslog master will not compile without > libfastjson. > > with mmnormalize, you can now have the json data type, so you can have > > rule=:%something:word% %foo:json%%bar:word% to deal with json embedded in > the middle of a log message. Given the performance attention that > liblognorm has had in the last year, I would not be surprised to find this > at least as fast as mmjsonparse. mmnormalize also lets you match the raw > logs, so if you have something (like logstash grumble grumble) that really > doesn't want to talk traditional syslog, but only json chunks, you can > parse the raw logs with > > rule=:%.:json% > > and everything will show up and 'just work' :-) > > David Lang > > > On Thu, 4 Feb 2016, Bob Gregory wrote: > > >> 1. I wasn't aware that mmnormalize is now json capable. I might kick the >> tyres on that, but the lookup works for me thus far. >> 2. I'm currently building from master because I've got a PR open for >> version 8.17 >> 3. IIRC master is currently building against json-c - is that true? In >> either case, where do I find more info on libjsonfast? Google tells me >> nothing. >> >> - B >> >> On 4 February 2016 at 17:02, David Lang <[email protected]> wrote: >> >> On Thu, 4 Feb 2016, Bob Gregory wrote: >>> >>> Hi Dave, >>> >>>> >>>> It's the latter. Currently docker is just spraying logs out onto disk, >>>> in >>>> both plain text and json format, and there's no logrotate. Instead, we >>>> want >>>> just the json logs to go through rsyslog. We'll forward INFO level >>>> application logs to Elasticsearch via Redis, and put a human-readable >>>> version of logs into the journal. >>>> >>>> Marking the journal entries with the appropriate syslog severity makes >>>> it >>>> easy to query and filter. >>>> >>>> The lookup_table functionality actually works better than my proposed >>>> property replacer, because it's simple to modify the lookup if >>>> requirements >>>> evolve. >>>> >>>> >>> a couple comments >>> >>> 1. using mmnormalize and the latest liblognorm (with the version=2 >>> ruleset), rsyslog can parse raw json, it doesn't need the @cee token any >>> longer and can parse logs that are a mix of json and non-json data. >>> >>> 2. the table_lookup code that is in the released versions of rsyslog is >>> very limited and has some known bugs. It was a prototype from work that >>> was >>> discussed and was going to be sponsored, but the company initiating the >>> work fell through. Yesterday a full implementation was merged into the >>> master tree for release in 8.17. You really will want to be using that >>> version for anything beyond a proof of concept. >>> >>> 3. we have found some nasty bugs in the json-c library and as a result >>> have forked it to libjsonfast, 8.16 will optionally use it if it's >>> available, 8.17 will require it. >>> >>> and 8.17 (or a daily build version of it) will pull in the latest >>> liblognorm and libjsonfast. >>> >>> This is one of those cases where you will really want to be on the very >>> latest version. >>> >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> >> >> >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- ---- *Bob Gregory* Application Architect MADE.COM <http://www.made.com/> Skype: flinkywistypomm [image: MADE] Made.com Design Limited is a company registered in England and Wales. Registered number: 07101408 | Registered office: 100 Charing Cross Road, London WC2H 0HG _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
