Hi Mike, Thank you for the input! I’ve also used the ‘Logger’ field in the encoder to get the logger name – which is the differentiator fields concatted together.
Regards, Satish From: Mike Gibson [mailto:[email protected]] Sent: Friday, August 05, 2016 5:33 PM To: Satish Anupindi Rao Cc: [email protected] Subject: Re: [heka] Heka input differentiator fields to decoder Hi Satish, You may be able to achieve what you are asking by using the ScribbleDecoder. http://hekad.readthedocs.io/en/latest/config/decoders/scribble.html In combination with your SomethingLogDecoder both within a MultiDecoder. i.e. [SomethingLogInput] type = "LogstreamerInput" log_directory = "/var/log/xxx" file_match = '(?P<something>\s+)/program\.log\.?(?P<Seq>\d*)' differentiator = ["something", "program","Seq"] decoder = "SomethingLogDecoder" [SomethingLogMultiDecoder] type = "MultiDecoder" subs = ["SomethingLogDecoder", "SomethingLogScribbler"] cascade_strategy = "all" log_sub_errors = true [SomethingLogDecoder] type = "PayloadRegexDecoder" match_regex = '(?P<comp>[^(]+\w+)\@(?P<host>\D+[^)]).*node[(]+.*\.' [SomethingLogDecoder.message_fields] Type = "erlang" comp = '%comp%' host = '%host%' seq = '%Seq%' logger = '%Component%' [SomethingLogScribbler] type = "ScribbleDecoder" [SomethingLogScribbler.message_fields] Something = "something" On Thu, Aug 4, 2016 at 7:08 PM, Satish Anupindi Rao <[email protected]<mailto:[email protected]>> wrote: Hi, Please allow me to rephrase my question. It may not have been clear the first time. I have many folders as below with files as shown. /var/log/xxx/jack/program.log.1 /var/log/xxx/jack/program.log.2 /var/log/xxx/john/program.log.1 /var/log/xxx/john/program.log.3 /var/log/xxx/sam/program.log.1 ... The LogStreamerInput is shown below and I have tested it – it works just fine. [SomethingLogInput] type = "LogstreamerInput" log_directory = "/var/log/xxx" file_match = '(?P<something>\s+)/program\.log\.?(?P<Seq>\d*)' differentiator = ["something", "program","Seq"] decoder = "SomethingLogDecoder" The SomethingLogDecoder is below and I have verified that it works for the fields which are in the match_regex i.e comp and host [SomethingLogDecoder] type = "PayloadRegexDecoder" match_regex = '(?P<comp>[^(]+\w+)\@(?P<host>\D+[^)]).*node[(]+.*\.' [SomethingLogDecoder.message_fields] Type = "erlang" comp = '%comp%' host = '%host%' seq = '%Seq%' logger = '%Component%' I would like the ElasticSearchJSONEncoder to have access to the “something” field which was used as a differentiator in the Input phase. I need it since it gives me information if it was the “jack” or “john” or “sam” folder the messages came from. But I have not been able to figure out a way to do this i.e. have the encoder include fields from the input phase. Would anyone be able to help me out here? Thanks! From: Heka [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Satish Anupindi Rao Sent: Thursday, August 04, 2016 1:28 PM To: [email protected]<mailto:[email protected]> Subject: [heka] Heka input differentiator fields to decoder Hi! I have many folders with files as follows : /var/log/xxx/jack/program.log.1 /var/log/xxx/jack/program.log.2 /var/log/xxx/john/program.log.1 /var/log/xxx/john/program.log.3 /var/log/xxx/sam/program.log.1 ... I would like to pass the "something" field to the decoder , so that the message fields of the decoder also have access to the "something" fields. Is there a way for me to do this ? I post these fields via a REST api (using an elasticsearchoutput) and store them in a mongo DB. I would like the "something" field also stored. [SomethingLogInput] type = "LogstreamerInput" log_directory = "/var/log/xxx" file_match = '(?P<something>\s+)/program\.log\.?(?P<Seq>\d*)' differentiator = ["something", "erlang","Seq"] decoder = "SomethingLogDecoder" Thanks! _______________________________________________ Heka mailing list [email protected]<mailto:[email protected]> https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
