From what I can tell (and it's not very clear), it looks like you've
got one Heka instance running that has only a TcpInput, nothing else.
That will accept data, but it's not going to do anything with that data.
Then you've got a separate Heka config that contains no inputs, but only
a TcpOutput (pointing at the input that's specified in the other config)
and a FileOutput. These outputs might conceivably send data somewhere,
but there are no inputs, so it's not clear where that data would come from.
Drop the TcpOutput altogether, and combine the TcpInput and the
FileOutput into a single config:
[hekad]
maxprocs = 1
share_dir = "/Users/egu/heka/share/heka"
[tcp_in:3242]
type = "TcpInput"
splitter = "HekaFramingSplitter"
decoder = "ProtobufDecoder"
address = ":3242"
[tcp_heka_output_log]
type = "FileOutput"
message_matcher = "TRUE"
path = "/tmp/output.log"
perm = "664"
encoder = "tcp_heka_output_encoder"
[tcp_heka_output_encoder]
type = "PayloadEncoder"
append_newlines = false
Once you've done that, you should be able to use `heka-inject` to send a
message into your running Heka:
$ heka-inject -heka 127.0.0.1:3242 -payload "1212 this is just a test"
If you want to send custom data in through that TcpInput, then you'll
have to switch to using a different splitter and a different decoder,
the default setup you're using will only know how to handle Heka
protobuf streams.
-r
On 01/07/2016 03:48 PM, Emily Gu wrote:
Thanks you both Rob and David very much!
Not sure where I need to define "base_dir"?
I'm going to write a Heka plugin to pass our metrics data into Heka.
For now, I have a hard time to see the data I send in through
TCP programmatically through TcpInput in the output.log file.
I don't see any output. The configs are:
tcp_input.toml
============
[hekad]
maxprocs = 1
share_dir = "/Users/egu/heka/share/heka"
[tcp_in:3242]
type = "TcpInput"
splitter = "HekaFramingSplitter"
decoder = "ProtobufDecoder"
address = ":3242"
tcp_output.toml
==============
[hekad]
maxprocs = 1
share_dir = "/Users/egu/heka/share/heka"
[tcp_out:3242]
type = "TcpOutput"
message_matcher = "TRUE"
address = "127.0.0.1:3242 <http://127.0.0.1:3242>"
[tcp_heka_output_log]
type = "FileOutput"
message_matcher = "TRUE"
path = "/tmp/output.log"
perm = "664"
encoder = "tcp_heka_output_encoder"
[tcp_heka_output_encoder]
type = "PayloadEncoder"
append_newlines = false
The client:
package main
import (
"fmt"
"github.com/mozilla-services/heka/client
<http://github.com/mozilla-services/heka/client>"
)
func main() {
message_bytes := []byte {100}
sender, err := client.NewNetworkSender("tcp", "127.0.0.1:3242
<http://127.0.0.1:3242>")
if err != nil {
fmt.Println("Could not connect to", "127.0.0.1:3242
<http://127.0.0.1:3242>")
return
}
fmt.Println("Connected")
var i int
for i = 0; i < 10; i++ {
fmt.Println("message byte:", string(message_bytes))
err = sender.SendMessage(message_bytes)
if err != nil {
break
}
}
fmt.Println("sent", i, "messages")
}
Please let me know what else I need to change.
Thanks,
Emily
On Thu, Jan 7, 2016 at 3:28 PM, David Birdsong <[email protected]
<mailto:[email protected]>> wrote:
On Thu, Jan 7, 2016 at 3:22 PM, Rob Miller <[email protected]
<mailto:[email protected]>> wrote:
On 01/07/2016 03:09 PM, Emily Gu wrote:
Thanks David for all the help! I'll give it a try.
Please bear with me as some parts I still not understand.
1. Why do I have to run two Heka instances where one for
input and
another for output?
Because if you send the output from a Heka instance back into
itself, then you're likely setting up an infinite loop of
traffic that will spin out of control.
2. Did you mean I need to specify different share_dirs in
input and
output Toml configs?
If you're running multiple Heka instances on a single machine,
it *should* be fine for them to use the same share_dir, which is
read-only. It's very important that each specifies a unique
base_dir, however, since that's used by Heka for internal
bookkeeping data. Two Heka's using the same base_dir is asking
for trouble.
3. Do I need both TcpOutput and FileOutput in order for me
to see
messages inside an output file? What if I didn't specify
TcpOutput?
Um, TcpOutput sends output data over a TCP connection. It
expects that there is a listener on the other side which will
accept that TCP connection, and will know how to correctly
handle the data that Heka is sending over the TCP connection.
FileOutput sends data to a file on the local file system.
It's of course fine to specify a FileOutput without specifying a
TcpOutput.
-r
whoops, yes I meant base_dir for where heka writes various internal
state information to.
Emily,
Maybe you could share what data you're trying to read into heka and
what you would like to do with it and we could help get you going.
Heka intended to a uni-directional pipeline. It can read data in
from many places into various formats, aggregate into interesting
new formats, and finally emit data somewhere else.
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
