Thanks for the point, David. I'll take a look. Thanks, Emily
On Thu, Jan 7, 2016 at 4:07 PM, David Birdsong <[email protected]> wrote: > here's a shorter, more succinct gist: > https://gist.github.com/davidbirdsong/e2a829c9519790e8d9df > > On Thu, Jan 7, 2016 at 4:04 PM, David Birdsong <[email protected]> > wrote: > >> great, that's the info we needed. >> >> so you can drop the TcpOutput for now since it appears that you are >> trying to write in the native heka format to a heka process from your own >> app. having heka write to another endpoint might be useful later, but it >> doesn't need to write to itself or the LogOutput plugin to display your >> messages in stdout. >> >> i'm not sure what it's in message_bytes, but here's a snippet that you >> can use as a reference. >> >> https://github.com/imgix/hekametrics/blob/master/hekalogger.go >> >> >> On Thu, Jan 7, 2016 at 3:48 PM, Emily Gu <[email protected]> wrote: >> >>> Thanks you both Rob and David very much! >>> >>> Not sure where I need to define "base_dir"? >>> >>> I'm going to write a Heka plugin to pass our metrics data into Heka. >>> >>> For now, I have a hard time to see the data I send in through >>> TCP programmatically through TcpInput in the output.log file. >>> I don't see any output. The configs are: >>> >>> tcp_input.toml >>> ============ >>> >>> [hekad] >>> >>> maxprocs = 1 >>> >>> share_dir = "/Users/egu/heka/share/heka" >>> >>> >>> [tcp_in:3242] >>> >>> type = "TcpInput" >>> >>> splitter = "HekaFramingSplitter" >>> >>> decoder = "ProtobufDecoder" >>> >>> address = ":3242" >>> >>> >>> tcp_output.toml >>> >>> ============== >>> >>> [hekad] >>> >>> maxprocs = 1 >>> >>> share_dir = "/Users/egu/heka/share/heka" >>> >>> >>> [tcp_out:3242] >>> >>> type = "TcpOutput" >>> >>> message_matcher = "TRUE" >>> >>> address = "127.0.0.1:3242" >>> >>> >>> [tcp_heka_output_log] >>> >>> type = "FileOutput" >>> >>> message_matcher = "TRUE" >>> >>> path = "/tmp/output.log" >>> >>> perm = "664" >>> >>> encoder = "tcp_heka_output_encoder" >>> >>> >>> [tcp_heka_output_encoder] >>> >>> type = "PayloadEncoder" >>> >>> append_newlines = false >>> >>> >>> The client: >>> >>> package main >>> >>> >>> import ( >>> >>> "fmt" >>> >>> "github.com/mozilla-services/heka/client" >>> >>> ) >>> >>> >>> >>> func main() { >>> >>> message_bytes := []byte {100} >>> >>> >>> sender, err := client.NewNetworkSender("tcp", "127.0.0.1:3242") >>> >>> if err != nil { >>> >>> fmt.Println("Could not connect to", "127.0.0.1:3242") >>> >>> return >>> >>> } >>> >>> fmt.Println("Connected") >>> >>> >>> >>> var i int >>> >>> for i = 0; i < 10; i++ { >>> >>> fmt.Println("message byte:", string(message_bytes)) >>> >>> err = sender.SendMessage(message_bytes) >>> >>> if err != nil { >>> >>> break >>> >>> } >>> >>> } >>> >>> fmt.Println("sent", i, "messages") >>> >>> } >>> >>> >>> Please let me know what else I need to change. >>> >>> Thanks, >>> >>> Emily >>> >>> >>> >>> >>> >>> >>> On Thu, Jan 7, 2016 at 3:28 PM, David Birdsong <[email protected] >>> > wrote: >>> >>>> >>>> >>>> On Thu, Jan 7, 2016 at 3:22 PM, Rob Miller <[email protected]> wrote: >>>> >>>>> On 01/07/2016 03:09 PM, Emily Gu wrote: >>>>> >>>>>> Thanks David for all the help! I'll give it a try. >>>>>> >>>>>> Please bear with me as some parts I still not understand. >>>>>> >>>>>> 1. Why do I have to run two Heka instances where one for input and >>>>>> another for output? >>>>>> >>>>> >>>>> Because if you send the output from a Heka instance back into itself, >>>>> then you're likely setting up an infinite loop of traffic that will spin >>>>> out of control. >>>>> >>>>> 2. Did you mean I need to specify different share_dirs in input and >>>>>> output Toml configs? >>>>>> >>>>> >>>>> If you're running multiple Heka instances on a single machine, it >>>>> *should* be fine for them to use the same share_dir, which is read-only. >>>>> It's very important that each specifies a unique base_dir, however, since >>>>> that's used by Heka for internal bookkeeping data. Two Heka's using the >>>>> same base_dir is asking for trouble. >>>>> >>>>> 3. Do I need both TcpOutput and FileOutput in order for me to see >>>>>> messages inside an output file? What if I didn't specify TcpOutput? >>>>>> >>>>> >>>>> Um, TcpOutput sends output data over a TCP connection. It expects that >>>>> there is a listener on the other side which will accept that TCP >>>>> connection, and will know how to correctly handle the data that Heka is >>>>> sending over the TCP connection. >>>>> >>>>> FileOutput sends data to a file on the local file system. >>>>> >>>>> It's of course fine to specify a FileOutput without specifying a >>>>> TcpOutput. >>>>> >>>>> -r >>>>> >>>> >>>> whoops, yes I meant base_dir for where heka writes various internal >>>> state information to. >>>> >>>> Emily, >>>> >>>> Maybe you could share what data you're trying to read into heka and >>>> what you would like to do with it and we could help get you going. >>>> >>>> Heka intended to a uni-directional pipeline. It can read data in from >>>> many places into various formats, aggregate into interesting new formats, >>>> and finally emit data somewhere else. >>>> >>>> >>> >> >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
