If you have heka installed then you already have heka-inject, heka-flood, heka-cat, and the like. They're included in all of the binary packages, and automatically built for you when you build heka from source.
-r On 07/03/2015 12:14 PM, Henrik Feldt wrote:
Hi Rob, Thank you for your guidance. I'm trying to use heka-inject, but being a go-noob, I might be doing something wrong. I'm trying this: ~ go get github.com/mozilla-services/heka/cmd/heka-inject # github.com/mozilla-services/heka/message dev/gocode/src/github.com/mozilla-services/heka/message/message_matcher.go:21: undefined: tree This is the same error as I get when trying to import the client https://github.com/mozilla-services/heka/issues/1599 which I filed an issue for. I'm probably doing something wrong, or perhaps something funky was committed not too long ago to the repo? Regards, Henrik https://countermail.com/process.php?pubview=aGVucmlrQGxvZ2liaXQuc2U= On 3 Jul 2015, at 21:02, Rob Miller wrote: Sorry you're having trouble here. The docs you linked to are indeed correct. I'm afraid I don't have the cycles to dig in deeply to your code or the binary output that it's generating, but I can give you a couple of additional tips that might help with your debugging. My recommendation is that you start from a known good binary encoding of a Heka message stream and use that to compare your own output. Generating a binary stream is easy; if you set up a Heka instance with a FileOutput using a ProtobufEncoder then the generated file will contain exactly such a stream. The following steps should get you to something you can use as a reference: * Set up a testing Heka config w/ a TcpInput using a ProtobufDecoder, and a FileOutput using a ProtobufEncoder. * Use heka-inject (see https://hekad.readthedocs.org/en/latest/developing/testing.html#heka-inject) to generate a single Heka message and send it to the listening TcpInput. * Shut down Heka. As long as the message_matcher on the FileOutput was correctly set up to catch the message you sent, you should now have a file containing exactly one Heka message, along with the appropriate stream header. Then you can use the code that you're writing to generate a binary encoding for a message that is identical to the one you specified with the heka-inject call (i.e. matching payload, hostname, severity, type, etc. etc. etc.). Compare what your code is emitting with the binary stream in the file generated by Heka and see how they match up. Also, it might be useful to test your encoding logic separately from the TCP connection stuff, so bugs from the one aren't mistaken as problems with the other. To do this, I'd recommend starting out by having your code write the Heka message stream to disk. If this is done correctly, you will be able to see the contents of the message stream using the heka-cat tool (see https://hekad.readthedocs.org/en/latest/developing/testing.html#heka-cat). If your encoder is generating files that heka-cat can read, but you're still seeing I/O related errors, then you know the issue isn't with the stream format but is instead related to the I/O code itself. Sorry I can't be more help, hopefully this gives you enough to be able to debug and resolve your issue. -r On 07/03/2015 09:18 AM, Henrik Feldt wrote: Greetings, What is the binary format to send to Heka with? I'm trying to implement a client in F# for sending natively to Heka, but I'm failing. This is a trace: D 2015-07-03T15:22:12.9760700+00:00: running: writing to heka [Logary.Targets.Heka] write record separator write header length BAM!! System.IO.IOException: Write failure ---> System.Net.Sockets.SocketException: The socket has been shut down at System.Net.Sockets.Socket.Send (System.Byte[] buf, Int32 offset, Int32 size, SocketFlags flags) [0x00000] in <filename unknown>:0 at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0 //snip Here's the code: | Choice1Of2 (async { try use msgMs = msgMs use hdrMs = hdrMs printfn "write record separator" do s.WriteByte Constants.RecordSeparator do! Async.Sleep 200 printfn "write header length" do s.WriteByte (byte (hdrMs.Length)) do! Async.Sleep 200 printfn "write header" do! fromZero hdrMs (fun ms -> ms.CopyToAsync s) do! Async.Sleep 200 printfn "write unit separator" do s.WriteByte Constants.UnitSeparator do! Async.Sleep 200 printfn "write unit message" do! fromZero msgMs (fun ms -> ms.CopyToAsync s) with e -> printfn "BAM!! %A" e }) | Repro: https://github.com/logary/logary/tree/feature/heka/examples/heka or https://github.com/logary/logary/blob/feature/heka/src/targets/Logary.Targets.Heka/Script.fsx For the second repro above (.fsx), which contains more data, it looks like this: write record separator write header length write header BAM!! System.IO.IOException: EndWrite failure ---> System.Net.Sockets.SocketException: The socket has been shut down at System.Net.Sockets.Socket.EndSend (IAsyncResult result) [0x00000] in <filename unknown>:0 I have unit tests that mimick what your tests seem to have: * https://github.com/mozilla-services/heka/blob/dev/client/encoders_test.go * https://github.com/logary/logary/blob/feature/heka/src/tests/Logary.Targets.Heka.Tests/Program.fs#L28-L39 And I'm following this framing format: * https://hekad.readthedocs.org/en/latest/message/index.html So at this point I'm at my wits end, and can't find another way of doing it... Here's a sample byte array that I try to send to the socket (including the full message), on the wire: 1E02083E1F0A105C18DA1111A14D4DBAF1F8116DCC76CE1080C48C9491A9E8D4131A0B68656B612E6C6F676172792200320040BA8F024A0B636F696E64756374696F6E Here's the Heka config: |[hekad] maxprocs = 2 [TcpInput] address = ":5565" [PayloadEncoder] append_newlines = false [LogOutput] message_matcher = "TRUE" encoder = "PayloadEncoder" | The logs: 2015/07/03 09:10:06 Pre-loading: [DashboardOutput] 2015/07/03 09:10:06 Pre-loading: [ProtobufEncoder] 2015/07/03 09:10:06 Loading: [ProtobufEncoder] 2015/07/03 09:10:06 Pre-loading: [TokenSplitter] 2015/07/03 09:10:06 Loading: [TokenSplitter] 2015/07/03 09:10:06 Pre-loading: [HekaFramingSplitter] 2015/07/03 09:10:06 Loading: [HekaFramingSplitter] 2015/07/03 09:10:06 Pre-loading: [NullSplitter] 2015/07/03 09:10:06 Loading: [NullSplitter] 2015/07/03 09:10:06 Pre-loading: [ProtobufDecoder] 2015/07/03 09:10:06 Loading: [ProtobufDecoder] 2015/07/03 09:10:06 Loading: [DashboardOutput] 2015/07/03 09:10:06 Starting hekad... 2015/07/03 09:10:06 Output started: DashboardOutput 2015/07/03 09:10:06 MessageRouter started. I've verified I can connect using telnet to 5565. You can netcat it to Heka to test my code. Or you can use the |prepare.sh| script to test it live (you'll need to install mono first http://www.mono-project.com/download/ (or apt-get install mono-complete) Regards, Henrik https://countermail.com/process.php?pubview=aGVucmlrQGxvZ2liaXQuc2U= _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
