Yeah I know it's different because my input is LogStreamer and the example in docs was for receiving protobuf by tcp.
I want to understand. When my LogStreamer reads a message it passes to the decoder a Protocol Buffer message with the with the log line in the message payload, that's right? Using the following config as my input decoder (this is what I actually tried): [syslog-decoder] subs = ['nginx-access-decoder', 'ProtobufDecoder'] cascade_strategy = "first-wins" log_sub_errors = true [ProtobufDecoder] This should capture my nginx log lines and remove it from the decoding "cascade" and pass all the rest to ProtobufDecoder that in turn doesn't do nothing. Is this correct? And if it is, why is this so slow? On Wed, Feb 4, 2015 at 8:08 PM, Rob Miller <[email protected]> wrote: > The config that you cargo-culted from the docs is meant for an entirely > different use case. That's meant to handle cases where you're receiving > protocol buffer encoded Heka messages, each of which contains an Nginx > access log line as the message payload. This would be useful in a case > where one Heka is loading the log files but instead of parsing them it's > sending them along in protobuf format to another Heka that's doing the > parsing. The config below would be used on the listener. > > If you want to see the decoding errors all you need to do is change your > `log_sub_errors` setting from false to true. > > -r > > > > On 02/04/2015 04:19 AM, Victor Castell wrote: > >> I managed to get a working config but I want to understand what's going >> on: >> >> [syslog-decoder] >> type = "MultiDecoder" >> subs = ['nginx-access-decoder', 'rsyslog-decoder'] >> cascade_strategy = "first-wins" >> log_sub_errors = false >> >> In the nginx-access I'm deconding the corresponding access.log entries >> from my rsyslog and in the rsyslog decoder I'm capturing any other >> rsyslog entries and discarding them. >> >> That works well but in my first attempt I tried with the config >> extracted from the documentation: >> >> [shipped-nginx-decoder] >> type = "MultiDecoder" >> subs = ['ProtobufDecoder', 'nginx-access-decoder'] >> cascade_strategy = "all" >> log_sub_errors = true >> >> [ProtobufDecoder] >> >> I would rather this config than the previous one, because it can log me >> the errors of my nginx decoding. >> >> The problem is that when using the ProtobufDecoder the speed of decoding >> is really slow, and my nginx logs doesn't keep up with the current >> events in time, and it's always behind the current time. >> >> This doesn't happen with the rsyslog-decoder config, it parses the logs >> really fast. >> >> I thought it will be much faster using the internal ProtobufDecoder than >> a lua one but it's not the case. >> >> What's the reason for this? >> >> >> On Fri, Jan 30, 2015 at 11:31 AM, Victor Castell >> <[email protected] <mailto:[email protected]>> wrote: >> >> Didn't know of that! Life saver >> >> Thanks! >> >> El 30/1/2015 11:17, "Krzysztof Krzyżaniak" <[email protected] >> <mailto:[email protected]>> escribió: >> >> W dni pią 30 sty, 2015 o 10∶34 użytkownik Victor Castell >> <[email protected] <mailto:[email protected]>> >> napisał: >> > Hi! >> > >> > I have a centralized rsyslog formatted logfile and I'm >> > extracting nginx logs from there using heka and the nginx >> > access log decoder. >> > >> > The problem is that the parser also logs every other log >> > message out to heka.log. >> > >> > The volume of non nginx logs mixed in my rsyslog log is really >> > huge so heka.log file is growing like crazy (I have >> > logrotating before you ask) >> > >> > Is there a way to conditionally/intentionally suppress the >> > parsing errors of a given decoder? >> >> You probably want to use MultiDecoder which split nginx logs >> from the rest and use log_sub_errors = false in MultiDecoder >> section. >> >> eloy >> >> >> >> >> -- >> V >> >> >> _______________________________________________ >> Heka mailing list >> [email protected] >> https://mail.mozilla.org/listinfo/heka >> >> > -- V
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
