Oops, lost this one in the holiday shuffle, sorry. Maybe you've figured it out or routed around the problem by now, but I'll answer anyway.
The issue is that ProtobufDecoder expects to find its raw data in the pack's MsgBytes buffer, whereas most decoders expect to find the data in an already created message's Payload field. TcpInput defaults to using the HekaFramingSplitter, which always puts its output data into MsgBytes, but HttpListenInput defaults to a NullSplitter, which does not. NullSplitter is the right choice, you just have to tell the NullSplitter to put the data where the ProtobufDecoder expects it. Replacing your HttpListenInput stanza with the following should work: [null_use_msgbytes] type = "NullSplitter" use_message_bytes = true [HttpListenInput] address = "0.0.0.0:8325" decoder = "ProtobufDecoder" splitter = "null_use_msgbytes" -r On 12/14/2015 02:09 AM, Timur Batyrshin wrote:
Hi, Is it possible to send Heka messages in protobuf over HTTP? I’ve tried to do so using HTTPOutput and HTTPListenInput plugins each set with ProtobufEncoder/ProtobufDecoder but receiving heka only shows the following messages (which I assume mean that protobuf decoding has failed): 2015/12/14 10:00:36 :Timestamp: 1970-01-01 00:00:00 +0000 UTC :Type: :Hostname: :Pid: 0 :Uuid: :Logger: :Payload: :EnvVersion: :Severity: 7 If I change HTTPOutput+HTTPListenInput into TcpOutput/TcpInput (with no other config changes) all starts working fine. I need to use HTTP here but not plain TCP. I could use custom JSON Encoder instead of ProtobufEncoder but first I’d like to avoid extra conversion steps if possible. (PayloadEncoder would not work here as it doesn’t encode Hostname and the rest of standard Heka fields). Here are my configs for reference: # Sender [hekad] maxprocs = 2 poolsize = 100 plugin_chansize = 30 [DashboardOutput] ticker_interval = 5 message_matcher = "Hostname == 'ip-10-10-9-99' && (Type == 'heka.all-report' || Type == 'heka.sandbox-output' || Type == 'heka.sandbox-terminated')" [http_output] type = "HttpOutput" address = "http://54.229.73.31:8325"; encoder = "ProtobufEncoder" message_matcher = "TRUE" [hekad] maxprocs = 2 poolsize = 100 plugin_chansize = 30 [DashboardOutput] ticker_interval = 5 message_matcher = "TRUE" [HttpListenInput] address = "0.0.0.0:8325" decoder = "ProtobufDecoder" [debug_encoder] type="RstEncoder" [LogOutput] encoder = "debug_encoder" message_matcher = "Hostname != 'ip-10-10-9-156’" Both Hekas are 0.10.0b2 running on Ubuntu 12.04 on AWS. Thanks, Timur _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
