We have no intention of supporting grok in Heka. Yes, there are a lot of folks using grok out there. But grok was originally a layer built on top of regular expressions, designed to overcome the composability limitations that are there for standard regex. LPEG doesn't have these composability issues, it already has all of the flexibility of grok and then some. And it's leaner, so I imagine LPEG performance is much better than what you'll see with grok.
That being said, since there *is* an LPEG implementation of grok, there's nothing preventing anyone from using grok in Heka. You can drop that module in your module directory and start using it immediately. I would also be open to landing a SandboxDecoder implementation that uses the grok module to provide support for using grok parsing patterns out of the box. But since we (i.e. the Heka core team) aren't using grok ourselves, someone else will have to step up to the plate to actually do the initial work. -r On 09/24/2015 05:21 AM, Andre wrote:
Timur, No doubts, LPEG is awesome! So awesome that Lunatic's grok implementation in grok is all based around it: https://github.com/arekinath/loglunatic/blob/master/lunatic/filters/common.lua However the real advantage of grok is its user base and is reasonably broad support by tools (.e.g. LogStash, FluentD, Graylog2, and others)... This mean you gain an easy source of pre-defined parsing rules without having to rewrite them as a sandbox LPEG, e.g: https://github.com/whyscream/postfix-grok-patterns https://github.com/elastic/logstash/tree/v1.4.2/patterns http://www.gregmefford.com/blog/2014/09/24/analyzing-cisco-asa-firewall-logs-with-logstash/ http://seclists.org/snort/2013/q4/174 Another plus is that allows heka users to maintain a single patter base and process logs in a similar way across multiple platforms. I have been doing that with a legacy app using a PHP implemention of grok and it is so handy... You just copy the pattern folders from logstash to phpgrok and voila! Cheers On Thu, Sep 24, 2015 at 9:58 PM, Timur Batyrshin <[email protected]> wrote: > Hi Andre, > > Have you checked LPEG? It is really cool too. > Here is a tutorial for it: http://lua-users.org/wiki/LpegTutorial > Some real usage example in Heka can be used at > https://github.com/mozilla-services/heka/tree/dev/sandbox/lua/decoders > (for example > https://github.com/mozilla-services/heka/blob/dev/sandbox/lua/decoders/linux_loadavg.lua#L48-L58 > which is failrly easy to > understand even without reading docs on LPEG). > There is also an online LPEG testing tool as well with a few examples to > play with it easily: http://lpeg.trink.com/ > > Regards > Timur > > On 24 Sep 2015 at 13:38:54, Andre ([email protected]) wrote: > > Hi there, > > Grok is perhaps one of the coolest features of logstash (once you get > used to debug it...) > > Do you think heka should have similar capability? > > If yes, what would you reckon, Go or Lua?* > > > Cheers > > > * - I could find pre existing implementations of Grok in both languages > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
