Thanks you both Rob and David very much!
Not sure where I need to define "base_dir"?
I'm going to write a Heka plugin to pass our metrics data into Heka.
For now, I have a hard time to see the data I send in through
TCP programmatically through TcpInput in the output.log file.
I don't see any output. The configs are:
tcp_input.toml
============
[hekad]
maxprocs = 1
share_dir = "/Users/egu/heka/share/heka"
[tcp_in:3242]
type = "TcpInput"
splitter = "HekaFramingSplitter"
decoder = "ProtobufDecoder"
address = ":3242"
tcp_output.toml
==============
[hekad]
maxprocs = 1
share_dir = "/Users/egu/heka/share/heka"
[tcp_out:3242]
type = "TcpOutput"
message_matcher = "TRUE"
address = "127.0.0.1:3242"
[tcp_heka_output_log]
type = "FileOutput"
message_matcher = "TRUE"
path = "/tmp/output.log"
perm = "664"
encoder = "tcp_heka_output_encoder"
[tcp_heka_output_encoder]
type = "PayloadEncoder"
append_newlines = false
The client:
package main
import (
"fmt"
"github.com/mozilla-services/heka/client"
)
func main() {
message_bytes := []byte {100}
sender, err := client.NewNetworkSender("tcp", "127.0.0.1:3242")
if err != nil {
fmt.Println("Could not connect to", "127.0.0.1:3242")
return
}
fmt.Println("Connected")
var i int
for i = 0; i < 10; i++ {
fmt.Println("message byte:", string(message_bytes))
err = sender.SendMessage(message_bytes)
if err != nil {
break
}
}
fmt.Println("sent", i, "messages")
}
Please let me know what else I need to change.
Thanks,
Emily
On Thu, Jan 7, 2016 at 3:28 PM, David Birdsong <[email protected]>
wrote:
>
>
> On Thu, Jan 7, 2016 at 3:22 PM, Rob Miller <[email protected]> wrote:
>
>> On 01/07/2016 03:09 PM, Emily Gu wrote:
>>
>>> Thanks David for all the help! I'll give it a try.
>>>
>>> Please bear with me as some parts I still not understand.
>>>
>>> 1. Why do I have to run two Heka instances where one for input and
>>> another for output?
>>>
>>
>> Because if you send the output from a Heka instance back into itself,
>> then you're likely setting up an infinite loop of traffic that will spin
>> out of control.
>>
>> 2. Did you mean I need to specify different share_dirs in input and
>>> output Toml configs?
>>>
>>
>> If you're running multiple Heka instances on a single machine, it
>> *should* be fine for them to use the same share_dir, which is read-only.
>> It's very important that each specifies a unique base_dir, however, since
>> that's used by Heka for internal bookkeeping data. Two Heka's using the
>> same base_dir is asking for trouble.
>>
>> 3. Do I need both TcpOutput and FileOutput in order for me to see
>>> messages inside an output file? What if I didn't specify TcpOutput?
>>>
>>
>> Um, TcpOutput sends output data over a TCP connection. It expects that
>> there is a listener on the other side which will accept that TCP
>> connection, and will know how to correctly handle the data that Heka is
>> sending over the TCP connection.
>>
>> FileOutput sends data to a file on the local file system.
>>
>> It's of course fine to specify a FileOutput without specifying a
>> TcpOutput.
>>
>> -r
>>
>
> whoops, yes I meant base_dir for where heka writes various internal state
> information to.
>
> Emily,
>
> Maybe you could share what data you're trying to read into heka and what
> you would like to do with it and we could help get you going.
>
> Heka intended to a uni-directional pipeline. It can read data in from many
> places into various formats, aggregate into interesting new formats, and
> finally emit data somewhere else.
>
>
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
