Hi list :) I'm trying to understand if mmnormalize is a good fit for parsing a high traffic of logs, given the fact that events are really heterogeneous (think log4j logs, apache logs, whatever logs are commonly produced).
My only frame of reference is Logstash's grok filter<http://logstash.net/docs/1.2.2/filters/grok>, which allows you to tag regular expressions in a dictionary, and then use those tags to match fields from logs, and put them in a structured event. Much like how you'd build a liblognorm rulebase. If I got it right, the advantage of mmnormalize seems to be performance, because it goes around using regular expressions. Not sure how this actually work, though. Practically, it sounds like this comes at the expense of flexibility: if I need to add a new "pattern" in liblognorm (say, a new date format) I'd have to patch the library itself, no? If that's the case, it looks like grok would be more suitable for a heterogeneous environment, because you can just add/remove patterns at will. There's also a matter of popularity, because grok is quite widely used, so you can find ready-made dictionaries and rules quite easily. It's not only about Logstash, as Apache Flume uses a library called Morphlines which also implements grok: http://blog.cloudera.com/blog/2013/07/morphlines-the-easy-way-to-build-and-integrate-etl-apps-for-apache-hadoop/ Basically, my question is whether liblognorm/mmnormalize can be made flexible enough to handle the common logging formats out there, or is it scoped to be a performance-oriented thing for specific use-cases? Speaking of scope, can liblognorm be enhanced to support parsing multiline messages? This seems to be possible in grok: https://logstash.jira.com/browse/LOGSTASH-692 For me, it's important to understand whether I should put effort in working with mmnormalize and sponsor needed enhancements, or would sponsoring a new "mmgrok" module be a better idea for my use-case. Because it looks like grok is available as a C library as well: https://github.com/jordansissel/grok Best regards, Radu _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
