Hey Rob, that explains exactly what I wanted to say and how I understood heka. Even though my language was not quite as exact.
Nested structures like: 1. extract a field from a message 2. create a new field with the extracted one 3. lookup something according to the field …. n. act on the steps before They have to be organized as filter-plugins, where step ‘2.’ matches the outcome of step ‘1.’, ‘3.’ matches ‘2.’ and so on. ’n.’ might be an output plugin using a encoder somehow. If the filter plugin is using a lua script this script could even be dynamically loaded. That’s not true for pure golang plugins (would be nice though, but I assume that’s not that easy). Cheers Christian > On 06 Oct 2015, at 18:28, Rob Miller <[email protected]> wrote: > > Sorry, I don't quite understand the question. There are no "steps" in Heka. > Heka's pipeline is the following: > > - Raw data comes in via an Input plugin > - A Splitter plugin slices the raw data into discrete records > - A Decoder plugin maps the raw data to a Heka message structure; > MultiDecoder can be used to apply more than one Decoder to the same input data > - Messages are handed to the router > - Router checks the message_matcher for all Filter and Output plugins, > delivers the message to any that are a match > - Optionally, Filter plugins might generate new messages that will be dropped > onto the router > - Output plugin invokes an Encoder to turn the message back into a raw data > stream > - Output plugin sends data out > > If you explain what you're trying to accomplish at a high level, I could > probably point you in the right direction, but the current language that > you're using doesn't make much sense to me. > > -r > > > On 10/06/2015 02:06 AM, Christian Kniep wrote: >> Hey y'all, >> >> on that topic ‘kinda replacing Logstash’. >> >> I am not mistaken it’s possible to chain multiple steps together as one >> would do in logstash, correct? >> I use the [ProcStatDecoder] and pass it in a couple of steps towards the >> [LogOutput] by stating the previous step in the ‘message_type’. >> As far as I understood it (or hope it works) it will parse all the >> config and afterwards create the correct workflow based on the matcher >> and name of the steps. >> >> Cheers >> Christian >> >> >> -- >> >> >> >> Christian Kniep |Release Engineer >> >> www.gaikai.com <http://www.gaikai.com/> <http://www.gaikai.com/ >> <http://www.gaikai.com/>> >> >>> On 06 Oct 2015, at 10:32, Robert Gardam >>> <[email protected] >>> <mailto:[email protected]> >>> <mailto:[email protected] >>> <mailto:[email protected]>>> wrote: >>> >>> I managed to get it to work. I think the only way to make this work is >>> to use the multi-decoder as I still need all other syslog messages to be >>> processed in heka and passed onto es. >>> >>> >>> On 05/10/15 22:58, Rob Miller wrote: >>>> On 10/5/15 9:59 AM, Robert Gardam wrote: >>>>> Hi All, >>>>> I'm just trying out moving from logstash and I'm wondering how does one >>>>> push one message from one decoder into another decoder after it has been >>>>> decoded the first time. >>>>> >>>>> Essentially i'm decoding json rsyslog messages from a tcp input and then >>>>> I want to pass only one field into the nginx access log decoder. >>>> Most of the work for the nginx access decoder is done in a module, so >>>> I'd recommend just copying the nginx decoder code and adding your JSON >>>> parsing right in the same decoder. >>>>> I can multi decode, but this only seems to decode json first but not >>>>> pass the json message into the second decoder. >>>> MultiDecoder should work, albeit less efficiently. You have to set >>>> `cascade_strategy = "all"` (the default is "first-wins") for your use >>>> case, though; did you set that? >>>> >>>> -r >>>> >>>> _______________________________________________ >>>> Heka mailing list >>>> [email protected] <mailto:[email protected]> <mailto:[email protected] >>>> <mailto:[email protected]>> >>>> https://mail.mozilla.org/listinfo/heka >>>> <https://mail.mozilla.org/listinfo/heka> >>> >>> -- >>> Robert Gardam >>> DevOps Engineer >>> Native Instruments GmbH >>> +49-30-611035-1775 >>> >>> www.native-instruments.com <http://www.native-instruments.com/> >>> <http://www.native-instruments.com <http://www.native-instruments.com/>> >>> _______________________________________________ >>> Heka mailing list >>> [email protected] <mailto:[email protected]> >>> https://mail.mozilla.org/listinfo/heka >>> <https://mail.mozilla.org/listinfo/heka> >> >> >> >> _______________________________________________ >> Heka mailing list >> [email protected] <mailto:[email protected]> >> https://mail.mozilla.org/listinfo/heka >> <https://mail.mozilla.org/listinfo/heka> >> > _______________________________________________ > Heka mailing list > [email protected] <mailto:[email protected]> > https://mail.mozilla.org/listinfo/heka > <https://mail.mozilla.org/listinfo/heka>
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
