Even though I got a workaround by piping it through rsyslog: #### # cat /etc/rsyslog.d/heka.conf $template kafka,"%timestamp:::date-rfc3339% %msg%\n" *.* @@127.0.0.1:5514;kafka #### Turns out: #### 2015/08/19 23:38:07 2015-08-19T23:38:07.665732+02:00 Test7 ####
Which might be nice, since syslog is a good central entry point, even if I would like to replace syslog completely in my containers - if possible and stable for all logging clients. Cheers guys Christian > On 19 Aug 2015, at 23:32, Christian Kniep <[email protected]> wrote: > > Bonus question, but I don’t want to stress it to much… > > What is different if I send the string via ‘logger’? > #### > logger -n 127.0.0.1 -P 5514 -T Test6 > #### > This only shows the closing TCP connection but now decoding/splitting… > #### > 2015/08/19 23:32:20 Decoder 'TcpInput-newline_decoder-127.0.0.1': stopped > #### > > Good n8 from Berlin > Christian > >> On 19 Aug 2015, at 23:13, Nathan Williams <[email protected] >> <mailto:[email protected]>> wrote: >> >> i also noticed that that echo isn't actually inserting newlines, but literal >> '\n' instead. fwiw, it works a bit better using printf instead. >> >> On Wed, Aug 19, 2015 at 2:10 PM, Christian Kniep <[email protected] >> <mailto:[email protected]>> wrote: >> Cool, >> >> thanks guys… that seems to solve my issue! >> >> #### >> # cat /etc/heka/hekad.toml >> [hekad] >> maxprocs = 2 >> >> [TcpInput] >> address = ":5514" >> splitter = "newline_splitter" >> decoder = "better_decoder" >> >> [newline_splitter] >> type = "TokenSplitter" >> delimiter = '\n' >> >> [better_decoder] >> type = "ScribbleDecoder" >> [better_decoder.message_fields] >> Type = "MyType" >> >> [gross_decoder] >> type = "PayloadRegexDecoder" >> match_regex = "^(?P<Message>.+)" >> >> [PayloadEncoder] >> append_newlines = true >> >> [LogOutput] >> message_matcher = "TRUE" >> encoder = “PayloadEncoder" >> ##### >> >> Cheers >> Christian >> >> > On 19 Aug 2015, at 23:06, Rob Miller <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > What you're seeing is normal. When your client shuts down, the TCP >> > connection closes, so Heka spins down the goroutines for handling that >> > connection. Each connection gets its own decoder instance, so you're >> > seeing the ProtobufDecoder for that connection exiting. >> > >> > Since TcpInput defaults to using ProtobufDecoder, you'll need to specify >> > something to override the default. Ideally you'd do some actual parsing >> > here, but if you don't need to do any parsing the most minimal performance >> > impact would probably be to use a ScribbleDecoder to scribble some value >> > onto each message. >> > >> > -r >> > >> > >> > On 08/19/2015 01:44 PM, Christian Kniep wrote: >> >> Hey y’all, >> >> >> >> after opening an issue on github I am still not sure how to create the >> >> TCP input to allow me sending arbitrary strings. >> >> https://github.com/mozilla-services/heka/issues/1683 >> >> <https://github.com/mozilla-services/heka/issues/1683> >> >> >> >> I am creating a docker image out of it: >> >> https://github.com/qnib/docker-heka <https://github.com/qnib/docker-heka> >> >> >> >> My heka.toml looks as follows: >> >> ############## >> >> [hekad] >> >> maxprocs = 2 >> >> >> >> [TcpInput] >> >> address = ":5514" >> >> splitter = "newline_splitter" >> >> >> >> [newline_splitter] >> >> type = "RegexSplitter" >> >> delimiter = '\n' >> >> delimiter_eol = true >> >> >> >> [split_on_space] >> >> type = "TokenSplitter" >> >> delimiter = " " >> >> >> >> [PayloadEncoder] >> >> append_newlines = true >> >> >> >> [LogOutput] >> >> message_matcher = "TRUE" >> >> encoder = “PayloadEncoder" >> >> ############## >> >> >> >> If I send a string the TCPInput stops: >> >> ############## >> >> [root@heka tmp]# /opt/heka/bin/hekad -config=/etc/heka/hekad.toml >> >> 2015/08/19 22:38:14 Pre-loading: [LogOutput] >> >> 2015/08/19 22:38:14 Pre-loading: [TcpInput] >> >> 2015/08/19 22:38:14 Pre-loading: [newline_splitter] >> >> 2015/08/19 22:38:14 Pre-loading: [split_on_space] >> >> 2015/08/19 22:38:14 Pre-loading: [PayloadEncoder] >> >> 2015/08/19 22:38:14 Pre-loading: [HekaFramingSplitter] >> >> 2015/08/19 22:38:14 Loading: [HekaFramingSplitter] >> >> 2015/08/19 22:38:14 Pre-loading: [NullSplitter] >> >> 2015/08/19 22:38:14 Loading: [NullSplitter] >> >> 2015/08/19 22:38:14 Pre-loading: [ProtobufDecoder] >> >> 2015/08/19 22:38:14 Loading: [ProtobufDecoder] >> >> 2015/08/19 22:38:14 Pre-loading: [ProtobufEncoder] >> >> 2015/08/19 22:38:14 Loading: [ProtobufEncoder] >> >> 2015/08/19 22:38:14 Pre-loading: [TokenSplitter] >> >> 2015/08/19 22:38:14 Loading: [TokenSplitter] >> >> 2015/08/19 22:38:14 Loading: [PayloadEncoder] >> >> 2015/08/19 22:38:14 Loading: [newline_splitter] >> >> 2015/08/19 22:38:14 Loading: [split_on_space] >> >> 2015/08/19 22:38:14 Loading: [TcpInput] >> >> 2015/08/19 22:38:14 Loading: [LogOutput] >> >> 2015/08/19 22:38:14 Starting hekad... >> >> 2015/08/19 22:38:14 Output started: LogOutput >> >> 2015/08/19 22:38:14 MessageRouter started. >> >> 2015/08/19 22:38:14 Input started: TcpInput >> >> 2015/08/19 22:38:18 Decoder 'TcpInput-ProtobufDecoder-::1': stopped >> >> 2015/08/19 22:38:18 >> >> ############## >> >> While sending something like this: >> >> # echo "huhu\n"|nc -w1 localhost 5514 >> >> >> >> Cheers >> >> Christian >> >> >> >> >> >> _______________________________________________ >> >> Heka mailing list >> >> [email protected] <mailto:[email protected]> >> >> https://mail.mozilla.org/listinfo/heka >> >> <https://mail.mozilla.org/listinfo/heka> >> >> >> >> >> _______________________________________________ >> Heka mailing list >> [email protected] <mailto:[email protected]> >> https://mail.mozilla.org/listinfo/heka >> <https://mail.mozilla.org/listinfo/heka> >> >> >> _______________________________________________ >> Heka mailing list >> [email protected] <mailto:[email protected]> >> https://mail.mozilla.org/listinfo/heka > > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
