[adding the list back in] A quick tip you may or may not find helpful. All data in Heka will have to be conveyed in it's internal format before it can leave the Decode phase. Input and Decoders work closely together. Inputs usually handle socket level stuff, but they also have to figure out how to break a byte stream up into chunks to hand off to a decoder. Decoders look at the bytes chunks and convert what they can to a Heka message, dropping any byte chunks that don't match their expected encoding since they decode an encoding. One of the Input modules can read TCP data (another similar one reads UDP), it's parser_type config value is the thing that determines on what to break the byte stream up, it's decoder config value is the decoder that it hands those byte chunks directly to. The pastebin I sent you is the way you could configure a Tcp or Udp input module to read the native Heka format (which you could have other Heka's send data to.)
I wrote the go-metrics->heka connector which speaks the native Heka protocol since my aim was to preserve types across the wire. I'm currently writing a few production services and I'm standardizing on a transport and endpoint for those services. I haven't actually figured out what I'm going to do with the messages yet aside from forwarding them on to a datastore for now. When I get time, I'll look into how to use the cbufd filters and then some of the anomaly detection, but mostly I hope to hire somebody who can help me out with that sooner. The Fields fields in the Heka message seemd like the most sensible place to store various typed data, but it could be that I'm making my life harder--I doubt it though since it's so easy to leverage Heka further with Lua scripting. It sounds like you might want to send data from your app using Graphite's text line protocol. go-metrics can send in that format and Heka can read that format. I haven't used the StatsToFieldsDecoder decoder myself, but it appears to expect Graphite text lines. On Wed, Sep 17, 2014 at 6:41 PM, Ryan Richins <[email protected]> wrote: > Thanks for the help. I believe we were able to get the TcpInput to work > but it is when attempting to parse/decode the data after it is in Heka is > where the issues were today. She (coworker) tried to get the > StatsToFieldsDecoder to work but the decoder kept throwing an error about > invalid input (I don't have the exact error handy) > > Maybe it is just our lack of understanding of how the internals of heka > work but the end goal is to use the metrics generated from go-metrics and > get them into heka where they can be aggregated with the same metrics from > other instances of the same program. > > We will keep working on it and if you have any other suggestions please > throw them my way. > > Thanks again, > > Ryan > > > On Sep 17, 2014, at 9:06 PM, Ryan Richins <[email protected]> wrote: > > > > Begin forwarded message: > > *From: *David Birdsong <[email protected]> > *Subject: **Re: [heka] How to parse go-metrics data (fixed title)* > *Date: *September 17, 2014 8:54:35 PM EDT > *To: *Ryan Richins <[email protected]> > > i left a comment which you can add for the logoutput module to see all of > a message. > > On Wed, Sep 17, 2014 at 5:49 PM, David Birdsong <[email protected]> > wrote: > >> Here's the heka config: >> https://gist.github.com/davidbirdsong/248f61c33300cecc15c9 >> >> ...this works listener allows any other heka instances to send data or >> any heka client which encodes messages the same as a heka server to send. >> >> i haven't used the dashboard yet, but a simple logoutput should show >> values being populated into the "Fields" ...ie, not the payload. The fields >> are typed. >> >> Let me know how it goes. >> >> >> >> On Wed, Sep 17, 2014 at 5:43 PM, Ryan Richins <[email protected]> wrote: >> >>> David, >>> >>> Thanks for the help. I was tied up with a production issue all day but >>> a coworker tried to get heka to ingest the data from go-metrics using your >>> package but still could not get things working. Would you mind sharing a >>> minimal heka config that would work to ingest data from your package and >>> make the data available either in the heka dashboard or to a log output? >>> >>> Again, thanks for the help. >>> >>> Ryan Richins >>> >>> >>> >>> On Sep 16, 2014, at 10:22 PM, David Birdsong <[email protected]> >>> wrote: >>> >>> Ryan, I wrote a simple connector to send via the native Heka protobuf: >>> https://github.com/imgix/hekametrics >>> >>> Use is pretty simple. Let me know if you have any questions. >>> >>> On Tue, Sep 16, 2014 at 7:13 PM, Ryan Richins <[email protected]> >>> wrote: >>> >>>> Hi, >>>> >>>> I am writing a GO program that uses go-metrics ( >>>> https://github.com/rcrowley/go-metrics) to collect/provide application >>>> specific metrics. Our team is evaluating Heka and one of the first use >>>> cases is for each instance of the program to send its metric data to Heka >>>> but I am running into issues trying to get things working. The goal is to >>>> have multiple instances of the program each send metrics to Heka which >>>> would then aggregate the data and emit it to a downstream service. >>>> >>>> The go-metrics package can output metrics in various formats and the >>>> one I am trying to hook into Heka is Graphite ( >>>> https://github.com/rcrowley/go-metrics/blob/master/graphite.go). >>>> Looking through the Heka docs, there is the StatsToFieldsDecoder >>>> http://hekad.readthedocs.org/en/v0.7.1/config/decoders/index.html#statstofieldsdecoder >>>> which seems to be a perfect fit but the problem is figuring out which >>>> 'Input' to use. We spent some time trying to get the TcpInput to work >>>> (configuring go-metrics to connect to the TCP port Heka was listening on) >>>> but have not been successful yet. Is this the right path to go down to get >>>> data from go-metrics into Heka? >>>> >>>> Being a complete newbie to Heka, hopefully it is something simple that >>>> I am missing. >>>> >>>> Thanks, >>>> >>>> Ryan Richins >>>> _______________________________________________ >>>> Heka mailing list >>>> [email protected] >>>> https://mail.mozilla.org/listinfo/heka >>>> >>> >>> >>> >> > > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
