here's a shorter, more succinct gist: https://gist.github.com/davidbirdsong/e2a829c9519790e8d9df
On Thu, Jan 7, 2016 at 4:04 PM, David Birdsong <[email protected]> wrote: > great, that's the info we needed. > > so you can drop the TcpOutput for now since it appears that you are trying > to write in the native heka format to a heka process from your own app. > having heka write to another endpoint might be useful later, but it doesn't > need to write to itself or the LogOutput plugin to display your messages in > stdout. > > i'm not sure what it's in message_bytes, but here's a snippet that you can > use as a reference. > > https://github.com/imgix/hekametrics/blob/master/hekalogger.go > > > On Thu, Jan 7, 2016 at 3:48 PM, Emily Gu <[email protected]> wrote: > >> Thanks you both Rob and David very much! >> >> Not sure where I need to define "base_dir"? >> >> I'm going to write a Heka plugin to pass our metrics data into Heka. >> >> For now, I have a hard time to see the data I send in through >> TCP programmatically through TcpInput in the output.log file. >> I don't see any output. The configs are: >> >> tcp_input.toml >> ============ >> >> [hekad] >> >> maxprocs = 1 >> >> share_dir = "/Users/egu/heka/share/heka" >> >> >> [tcp_in:3242] >> >> type = "TcpInput" >> >> splitter = "HekaFramingSplitter" >> >> decoder = "ProtobufDecoder" >> >> address = ":3242" >> >> >> tcp_output.toml >> >> ============== >> >> [hekad] >> >> maxprocs = 1 >> >> share_dir = "/Users/egu/heka/share/heka" >> >> >> [tcp_out:3242] >> >> type = "TcpOutput" >> >> message_matcher = "TRUE" >> >> address = "127.0.0.1:3242" >> >> >> [tcp_heka_output_log] >> >> type = "FileOutput" >> >> message_matcher = "TRUE" >> >> path = "/tmp/output.log" >> >> perm = "664" >> >> encoder = "tcp_heka_output_encoder" >> >> >> [tcp_heka_output_encoder] >> >> type = "PayloadEncoder" >> >> append_newlines = false >> >> >> The client: >> >> package main >> >> >> import ( >> >> "fmt" >> >> "github.com/mozilla-services/heka/client" >> >> ) >> >> >> >> func main() { >> >> message_bytes := []byte {100} >> >> >> sender, err := client.NewNetworkSender("tcp", "127.0.0.1:3242") >> >> if err != nil { >> >> fmt.Println("Could not connect to", "127.0.0.1:3242") >> >> return >> >> } >> >> fmt.Println("Connected") >> >> >> >> var i int >> >> for i = 0; i < 10; i++ { >> >> fmt.Println("message byte:", string(message_bytes)) >> >> err = sender.SendMessage(message_bytes) >> >> if err != nil { >> >> break >> >> } >> >> } >> >> fmt.Println("sent", i, "messages") >> >> } >> >> >> Please let me know what else I need to change. >> >> Thanks, >> >> Emily >> >> >> >> >> >> >> On Thu, Jan 7, 2016 at 3:28 PM, David Birdsong <[email protected]> >> wrote: >> >>> >>> >>> On Thu, Jan 7, 2016 at 3:22 PM, Rob Miller <[email protected]> wrote: >>> >>>> On 01/07/2016 03:09 PM, Emily Gu wrote: >>>> >>>>> Thanks David for all the help! I'll give it a try. >>>>> >>>>> Please bear with me as some parts I still not understand. >>>>> >>>>> 1. Why do I have to run two Heka instances where one for input and >>>>> another for output? >>>>> >>>> >>>> Because if you send the output from a Heka instance back into itself, >>>> then you're likely setting up an infinite loop of traffic that will spin >>>> out of control. >>>> >>>> 2. Did you mean I need to specify different share_dirs in input and >>>>> output Toml configs? >>>>> >>>> >>>> If you're running multiple Heka instances on a single machine, it >>>> *should* be fine for them to use the same share_dir, which is read-only. >>>> It's very important that each specifies a unique base_dir, however, since >>>> that's used by Heka for internal bookkeeping data. Two Heka's using the >>>> same base_dir is asking for trouble. >>>> >>>> 3. Do I need both TcpOutput and FileOutput in order for me to see >>>>> messages inside an output file? What if I didn't specify TcpOutput? >>>>> >>>> >>>> Um, TcpOutput sends output data over a TCP connection. It expects that >>>> there is a listener on the other side which will accept that TCP >>>> connection, and will know how to correctly handle the data that Heka is >>>> sending over the TCP connection. >>>> >>>> FileOutput sends data to a file on the local file system. >>>> >>>> It's of course fine to specify a FileOutput without specifying a >>>> TcpOutput. >>>> >>>> -r >>>> >>> >>> whoops, yes I meant base_dir for where heka writes various internal >>> state information to. >>> >>> Emily, >>> >>> Maybe you could share what data you're trying to read into heka and what >>> you would like to do with it and we could help get you going. >>> >>> Heka intended to a uni-directional pipeline. It can read data in from >>> many places into various formats, aggregate into interesting new formats, >>> and finally emit data somewhere else. >>> >>> >> >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
