local installation worked. I'll look into why the docker installation of ES didn't work.
Thanks, Emily On Mon, Jan 11, 2016 at 3:41 PM, Emily Gu <[email protected]> wrote: > OK, thank you David. I'll see if we need that. > > Now I'm trying to publish heka message to elasticsearch, it got the > following error. > > *2016/01/11 15:18:02 Plugin 'ElasticSearchOutput' error: can't index: HTTP > request failed: Post http://192.168.100:9200/_bulk > <http://192.168.100:9200/_bulk>: dial tcp 192.168.0.100:9200 > <http://192.168.0.100:9200>: operation timed out* > > *configs are:* > > ====== > > [ESJsonEncoder] > > es_index_from_timestamp = true > > type_name = "%{Type}" > > > [ElasticSearchOutput] > > message_matcher = "Type == 'snap.heka'" > > server = "http://192.168.100:9200"; > > encoder = "ESJsonEncoder" > > flush_interval = 10 > > ======== > > *The message sample* > > *========* > > :Timestamp: 2016-01-11 23:38:01.411186108 +0000 UTC > > :Type: heka > > :Hostname: egu-mac01.lan > > :Pid: 68305 > > :Uuid: 9a26332d-64e7-4ff4-a7de-53a6a90e78b2 > > :Logger: heka.logger > > :Payload: > [{"namespace":["psutil","vm","free"],"last_advertised_time":"0001-01-01T00:00:00Z","version":0,"config":null,"data":35553280,"labels":null,"tags":null,"source":"egu-mac01.lan","timestamp":"2016-01-11T15:38:01.407168698-08:00"}] > > :EnvVersion: > > :Severity: 6 > > If you have any insight, please share. > > Thanks, > > Emily > > > > > > > On Mon, Jan 11, 2016 at 12:13 PM, David Birdsong <[email protected] > > wrote: > >> >> >> On Mon, Jan 11, 2016 at 12:07 PM, Emily Gu <[email protected]> wrote: >> >>> Thanks, David. I did this way. >>> >>> The collector is scheduled to collect metrics periodically. Collected >>> metrics are shipped to Heka in one shot. Then the connection is close. It >>> loops in the interval of the collectors(open connection -> ship metrics -> >>> close connection). Does this make sense? >>> >> >> yes, makes sense. >> >> >>> >>> I thought Heka has the pooling and queuing, it seems not necessary to >>> implement connection pooling in the client side. Same for the queuing on >>> the client side is not necessary. Please correct me if my understanding is >>> not correct. >>> >> >> no, connection pooling isn't necessary at all. the more messages you send >> over a single connection, the better, but if your pattern doesn't fit that, >> then it's fine. >> >> >>> >>> Thanks, >>> Emily >>> >>> >>> >>> On Mon, Jan 11, 2016 at 12:01 AM, David Birdsong < >>> [email protected]> wrote: >>> >>>> >>>> >>>> On Sun, Jan 10, 2016 at 9:46 PM, Emily Gu <[email protected]> wrote: >>>> >>>>> Hi Timur, >>>>> >>>>> Thanks for sharing your experience and data points. I got it working >>>>> now by using TCP. The performance has not been tested yet. I'll keep you >>>>> guys posted. For each message sent I can see one decoder stopped message >>>>> as >>>>> below. From what Rob explained, it seems to work as expected. >>>>> >>>> >>>> but not ideal. you should hold the tcp connection open and send many >>>> messages over a single connection. >>>> >>>> >>>>> *"2016/01/10 16:28:38 Decoder 'tcp_in:3242-ProtobufDecoder-127.0.0.1': >>>>> stopped"* >>>>> >>>>> Also, I can hear the pounding sound for each sent when volume is on. >>>>> I'm not sure if this is normal. >>>>> >>>>> thanks. >>>>> >>>>> Emily >>>>> >>>>> On Sun, Jan 10, 2016 at 2:12 AM, Timur Batyrshin <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Emily, >>>>>> >>>>>> Not sure what is your exact case for sending out data to Heka. >>>>>> Usually I find it much more easy to use JSON or similar plain text >>>>>> format to sending messages to Heka unless you have tight requirements for >>>>>> throughput. >>>>>> >>>>>> In my tests I've seen throughputs of ~1K messages/second (~10Mbit/s) >>>>>> on c4.large instance on AWS using stock lua JSON decoder/encoder and HTTP >>>>>> output/input. >>>>>> If you are expecting smaller throughputs you should probably look >>>>>> into that -- at least until you get used to Heka and to how it works. >>>>>> >>>>>> Best regards, >>>>>> Timur >>>>>> >>>>>> On Fri, Jan 8, 2016 at 3:22 AM, Emily Gu <[email protected]> wrote: >>>>>> >>>>>>> This is working. Thanks! >>>>>>> >>>>>>> I'm confusing on the two instances parts and also others. >>>>>>> >>>>>>> Yes, I need to send our custom data into Heka. I want to see if I >>>>>>> need to write my own custom Heka plugin or leverage existing Heka >>>>>>> plugins. >>>>>>> My custom data is a slice of metrics can send into Heka through TCP. >>>>>>> >>>>>>> Your suggestion is very much appreciated. >>>>>>> >>>>>>> Thanks, >>>>>>> Emily >>>>>>> >>>>>>> On Thu, Jan 7, 2016 at 4:10 PM, Rob Miller <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> From what I can tell (and it's not very clear), it looks like >>>>>>>> you've got one Heka instance running that has only a TcpInput, nothing >>>>>>>> else. That will accept data, but it's not going to do anything with >>>>>>>> that >>>>>>>> data. >>>>>>>> >>>>>>>> Then you've got a separate Heka config that contains no inputs, but >>>>>>>> only a TcpOutput (pointing at the input that's specified in the other >>>>>>>> config) and a FileOutput. These outputs might conceivably send data >>>>>>>> somewhere, but there are no inputs, so it's not clear where that data >>>>>>>> would >>>>>>>> come from. >>>>>>>> >>>>>>>> Drop the TcpOutput altogether, and combine the TcpInput and the >>>>>>>> FileOutput into a single config: >>>>>>>> >>>>>>>> [hekad] >>>>>>>> maxprocs = 1 >>>>>>>> share_dir = "/Users/egu/heka/share/heka" >>>>>>>> >>>>>>>> [tcp_in:3242] >>>>>>>> type = "TcpInput" >>>>>>>> splitter = "HekaFramingSplitter" >>>>>>>> decoder = "ProtobufDecoder" >>>>>>>> address = ":3242" >>>>>>>> >>>>>>>> [tcp_heka_output_log] >>>>>>>> type = "FileOutput" >>>>>>>> message_matcher = "TRUE" >>>>>>>> path = "/tmp/output.log" >>>>>>>> perm = "664" >>>>>>>> encoder = "tcp_heka_output_encoder" >>>>>>>> >>>>>>>> [tcp_heka_output_encoder] >>>>>>>> type = "PayloadEncoder" >>>>>>>> append_newlines = false >>>>>>>> >>>>>>>> >>>>>>>> Once you've done that, you should be able to use `heka-inject` to >>>>>>>> send a message into your running Heka: >>>>>>>> >>>>>>>> $ heka-inject -heka 127.0.0.1:3242 -payload "1212 this is just a >>>>>>>> test" >>>>>>>> >>>>>>>> If you want to send custom data in through that TcpInput, then >>>>>>>> you'll have to switch to using a different splitter and a different >>>>>>>> decoder, the default setup you're using will only know how to handle >>>>>>>> Heka >>>>>>>> protobuf streams. >>>>>>>> >>>>>>>> -r >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 01/07/2016 03:48 PM, Emily Gu wrote: >>>>>>>> >>>>>>>>> Thanks you both Rob and David very much! >>>>>>>>> >>>>>>>>> Not sure where I need to define "base_dir"? >>>>>>>>> >>>>>>>>> I'm going to write a Heka plugin to pass our metrics data into >>>>>>>>> Heka. >>>>>>>>> >>>>>>>>> For now, I have a hard time to see the data I send in through >>>>>>>>> TCP programmatically through TcpInput in the output.log file. >>>>>>>>> I don't see any output. The configs are: >>>>>>>>> >>>>>>>>> tcp_input.toml >>>>>>>>> ============ >>>>>>>>> >>>>>>>>> [hekad] >>>>>>>>> >>>>>>>>> maxprocs = 1 >>>>>>>>> >>>>>>>>> share_dir = "/Users/egu/heka/share/heka" >>>>>>>>> >>>>>>>>> >>>>>>>>> [tcp_in:3242] >>>>>>>>> >>>>>>>>> type = "TcpInput" >>>>>>>>> >>>>>>>>> splitter = "HekaFramingSplitter" >>>>>>>>> >>>>>>>>> decoder = "ProtobufDecoder" >>>>>>>>> >>>>>>>>> address = ":3242" >>>>>>>>> >>>>>>>>> >>>>>>>>> tcp_output.toml >>>>>>>>> >>>>>>>>> ============== >>>>>>>>> >>>>>>>>> [hekad] >>>>>>>>> >>>>>>>>> maxprocs = 1 >>>>>>>>> >>>>>>>>> share_dir = "/Users/egu/heka/share/heka" >>>>>>>>> >>>>>>>>> >>>>>>>>> [tcp_out:3242] >>>>>>>>> >>>>>>>>> type = "TcpOutput" >>>>>>>>> >>>>>>>>> message_matcher = "TRUE" >>>>>>>>> >>>>>>>>> address = "127.0.0.1:3242 <http://127.0.0.1:3242>" >>>>>>>>> >>>>>>>>> >>>>>>>>> [tcp_heka_output_log] >>>>>>>>> >>>>>>>>> type = "FileOutput" >>>>>>>>> >>>>>>>>> message_matcher = "TRUE" >>>>>>>>> >>>>>>>>> path = "/tmp/output.log" >>>>>>>>> >>>>>>>>> perm = "664" >>>>>>>>> >>>>>>>>> encoder = "tcp_heka_output_encoder" >>>>>>>>> >>>>>>>>> >>>>>>>>> [tcp_heka_output_encoder] >>>>>>>>> >>>>>>>>> type = "PayloadEncoder" >>>>>>>>> >>>>>>>>> append_newlines = false >>>>>>>>> >>>>>>>>> >>>>>>>>> The client: >>>>>>>>> >>>>>>>>> package main >>>>>>>>> >>>>>>>>> >>>>>>>>> import ( >>>>>>>>> >>>>>>>>> "fmt" >>>>>>>>> >>>>>>>>> "github.com/mozilla-services/heka/client >>>>>>>>> <http://github.com/mozilla-services/heka/client>" >>>>>>>>> >>>>>>>>> ) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> func main() { >>>>>>>>> >>>>>>>>> message_bytes := []byte {100} >>>>>>>>> >>>>>>>>> >>>>>>>>> sender, err := client.NewNetworkSender("tcp", "127.0.0.1:3242 >>>>>>>>> <http://127.0.0.1:3242>") >>>>>>>>> >>>>>>>>> if err != nil { >>>>>>>>> >>>>>>>>> fmt.Println("Could not connect to", "127.0.0.1:3242 >>>>>>>>> <http://127.0.0.1:3242>") >>>>>>>>> >>>>>>>>> return >>>>>>>>> >>>>>>>>> } >>>>>>>>> >>>>>>>>> fmt.Println("Connected") >>>>>>>>> >>>>>>>>> var i int >>>>>>>>> >>>>>>>>> for i = 0; i < 10; i++ { >>>>>>>>> >>>>>>>>> fmt.Println("message byte:", string(message_bytes)) >>>>>>>>> >>>>>>>>> err = sender.SendMessage(message_bytes) >>>>>>>>> >>>>>>>>> if err != nil { >>>>>>>>> >>>>>>>>> break >>>>>>>>> >>>>>>>>> } >>>>>>>>> >>>>>>>>> } >>>>>>>>> >>>>>>>>> fmt.Println("sent", i, "messages") >>>>>>>>> >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Please let me know what else I need to change. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Emily >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jan 7, 2016 at 3:28 PM, David Birdsong < >>>>>>>>> [email protected] >>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jan 7, 2016 at 3:22 PM, Rob Miller < >>>>>>>>> [email protected] >>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> On 01/07/2016 03:09 PM, Emily Gu wrote: >>>>>>>>> >>>>>>>>> Thanks David for all the help! I'll give it a try. >>>>>>>>> >>>>>>>>> Please bear with me as some parts I still not >>>>>>>>> understand. >>>>>>>>> >>>>>>>>> 1. Why do I have to run two Heka instances where one >>>>>>>>> for >>>>>>>>> input and >>>>>>>>> another for output? >>>>>>>>> >>>>>>>>> >>>>>>>>> Because if you send the output from a Heka instance back >>>>>>>>> into >>>>>>>>> itself, then you're likely setting up an infinite loop of >>>>>>>>> traffic that will spin out of control. >>>>>>>>> >>>>>>>>> 2. Did you mean I need to specify different share_dirs >>>>>>>>> in >>>>>>>>> input and >>>>>>>>> output Toml configs? >>>>>>>>> >>>>>>>>> >>>>>>>>> If you're running multiple Heka instances on a single >>>>>>>>> machine, >>>>>>>>> it *should* be fine for them to use the same share_dir, >>>>>>>>> which is >>>>>>>>> read-only. It's very important that each specifies a unique >>>>>>>>> base_dir, however, since that's used by Heka for internal >>>>>>>>> bookkeeping data. Two Heka's using the same base_dir is >>>>>>>>> asking >>>>>>>>> for trouble. >>>>>>>>> >>>>>>>>> 3. Do I need both TcpOutput and FileOutput in order >>>>>>>>> for me >>>>>>>>> to see >>>>>>>>> messages inside an output file? What if I didn't >>>>>>>>> specify >>>>>>>>> TcpOutput? >>>>>>>>> >>>>>>>>> >>>>>>>>> Um, TcpOutput sends output data over a TCP connection. It >>>>>>>>> expects that there is a listener on the other side which >>>>>>>>> will >>>>>>>>> accept that TCP connection, and will know how to correctly >>>>>>>>> handle the data that Heka is sending over the TCP >>>>>>>>> connection. >>>>>>>>> >>>>>>>>> FileOutput sends data to a file on the local file system. >>>>>>>>> >>>>>>>>> It's of course fine to specify a FileOutput without >>>>>>>>> specifying a >>>>>>>>> TcpOutput. >>>>>>>>> >>>>>>>>> -r >>>>>>>>> >>>>>>>>> >>>>>>>>> whoops, yes I meant base_dir for where heka writes various >>>>>>>>> internal >>>>>>>>> state information to. >>>>>>>>> >>>>>>>>> Emily, >>>>>>>>> >>>>>>>>> Maybe you could share what data you're trying to read into >>>>>>>>> heka and >>>>>>>>> what you would like to do with it and we could help get you >>>>>>>>> going. >>>>>>>>> >>>>>>>>> Heka intended to a uni-directional pipeline. It can read data >>>>>>>>> in >>>>>>>>> from many places into various formats, aggregate into >>>>>>>>> interesting >>>>>>>>> new formats, and finally emit data somewhere else. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Heka mailing list >>>>>>> [email protected] >>>>>>> https://mail.mozilla.org/listinfo/heka >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Heka mailing list >>>>> [email protected] >>>>> https://mail.mozilla.org/listinfo/heka >>>>> >>>>> >>>> >>> >> >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
