oops, libfastjson
https://github.com/rsyslog/libfastjson
http://blog.gerhards.net/2015/12/rsyslog-and-liblognorm-will-switch-to.html
I think the hard requirement for libfastjson just hit in the last day or so.
liblognorm will still use json-c (but will use libfastjson if it finds it), but
as of last night, rsyslog master will not compile without libfastjson.
with mmnormalize, you can now have the json data type, so you can have
rule=:%something:word% %foo:json%%bar:word% to deal with json embedded in the
middle of a log message. Given the performance attention that liblognorm has had
in the last year, I would not be surprised to find this at least as fast as
mmjsonparse. mmnormalize also lets you match the raw logs, so if you have
something (like logstash grumble grumble) that really doesn't want to talk
traditional syslog, but only json chunks, you can parse the raw logs with
rule=:%.:json%
and everything will show up and 'just work' :-)
David Lang
On Thu, 4 Feb 2016, Bob Gregory wrote:
1. I wasn't aware that mmnormalize is now json capable. I might kick the
tyres on that, but the lookup works for me thus far.
2. I'm currently building from master because I've got a PR open for
version 8.17
3. IIRC master is currently building against json-c - is that true? In
either case, where do I find more info on libjsonfast? Google tells me
nothing.
- B
On 4 February 2016 at 17:02, David Lang <[email protected]> wrote:
On Thu, 4 Feb 2016, Bob Gregory wrote:
Hi Dave,
It's the latter. Currently docker is just spraying logs out onto disk, in
both plain text and json format, and there's no logrotate. Instead, we
want
just the json logs to go through rsyslog. We'll forward INFO level
application logs to Elasticsearch via Redis, and put a human-readable
version of logs into the journal.
Marking the journal entries with the appropriate syslog severity makes it
easy to query and filter.
The lookup_table functionality actually works better than my proposed
property replacer, because it's simple to modify the lookup if
requirements
evolve.
a couple comments
1. using mmnormalize and the latest liblognorm (with the version=2
ruleset), rsyslog can parse raw json, it doesn't need the @cee token any
longer and can parse logs that are a mix of json and non-json data.
2. the table_lookup code that is in the released versions of rsyslog is
very limited and has some known bugs. It was a prototype from work that was
discussed and was going to be sponsored, but the company initiating the
work fell through. Yesterday a full implementation was merged into the
master tree for release in 8.17. You really will want to be using that
version for anything beyond a proof of concept.
3. we have found some nasty bugs in the json-c library and as a result
have forked it to libjsonfast, 8.16 will optionally use it if it's
available, 8.17 will require it.
and 8.17 (or a daily build version of it) will pull in the latest
liblognorm and libjsonfast.
This is one of those cases where you will really want to be on the very
latest version.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
