Re: Log4j/logback parser via syslog

[Bryan Bende]

I believe groovy, python, jython, jruby, ruby, javascript, and lua.

The associated JIRA is here:
https://issues.apache.org/jira/browse/NIFI-210

There are some cool blogs about them here:
http://funnifi.blogspot.com/2016/02/executescript-processor-hello-world.html

-Bryan

On Fri, Feb 12, 2016 at 10:48 AM, Madhukar Thota 
wrote:

> Thanks Bryan. I will look into ExtractText processor.
>
> Do you know what scripting languages are supported with new processors?
>
> -Madhu
>
> On Fri, Feb 12, 2016 at 9:27 AM, Bryan Bende  wrote:
>
>> Hello,
>>
>> Currently there are no built in processors to parse log formats, but have
>> you taken a look at the ExtractText processor [1]?
>>
>> If you can come up with a regular expression for whatever you are trying
>> to extract, then you should be able to use ExtractText.
>>
>> Other options...
>>
>> You could write a custom processor, but this sounds like it might be
>> overkill for your scenario.
>> In the next release (hopefully out in a few days) there will be two new
>> processors that support scripting languages. It may be easier to use a
>> scripting language to manipulate/parse the text.
>>
>> Thanks,
>>
>> Bryan
>>
>> [1]
>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
>>
>>
>> On Fri, Feb 12, 2016 at 12:16 AM, Madhukar Thota <
>> madhukar.th...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> I am very new to Apache Nifi and just started learning about how to use
>>> it.
>>>
>>> We have a requirement where we need to parse log4j/logback pattern
>>> messages coming from SyslogAppenders via Syslog udp. I can read the
>>> standard syslog messages, but how can i further extract log4j/logback
>>> messages  from syslog body.
>>>
>>> Is there any log parsers( log4j/logback/Apache access log format)
>>> available in apache nifi?
>>>
>>>
>>> Any help on this much appreciated.
>>>
>>> Thanks in Advance.
>>>
>>>
>>
>

Re: Log4j/logback parser via syslog

[Madhukar Thota]

Thanks Bryan. Looking forward for the release.



On Fri, Feb 12, 2016 at 10:55 AM, Bryan Bende  wrote:

> I believe groovy, python, jython, jruby, ruby, javascript, and lua.
>
> The associated JIRA is here:
> https://issues.apache.org/jira/browse/NIFI-210
>
> There are some cool blogs about them here:
>
> http://funnifi.blogspot.com/2016/02/executescript-processor-hello-world.html
>
> -Bryan
>
> On Fri, Feb 12, 2016 at 10:48 AM, Madhukar Thota  > wrote:
>
>> Thanks Bryan. I will look into ExtractText processor.
>>
>> Do you know what scripting languages are supported with new processors?
>>
>> -Madhu
>>
>> On Fri, Feb 12, 2016 at 9:27 AM, Bryan Bende  wrote:
>>
>>> Hello,
>>>
>>> Currently there are no built in processors to parse log formats, but
>>> have you taken a look at the ExtractText processor [1]?
>>>
>>> If you can come up with a regular expression for whatever you are trying
>>> to extract, then you should be able to use ExtractText.
>>>
>>> Other options...
>>>
>>> You could write a custom processor, but this sounds like it might be
>>> overkill for your scenario.
>>> In the next release (hopefully out in a few days) there will be two new
>>> processors that support scripting languages. It may be easier to use a
>>> scripting language to manipulate/parse the text.
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>> [1]
>>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html
>>>
>>>
>>> On Fri, Feb 12, 2016 at 12:16 AM, Madhukar Thota <
>>> madhukar.th...@gmail.com> wrote:
>>>
 Hi

 I am very new to Apache Nifi and just started learning about how to use
 it.

 We have a requirement where we need to parse log4j/logback pattern
 messages coming from SyslogAppenders via Syslog udp. I can read the
 standard syslog messages, but how can i further extract log4j/logback
 messages  from syslog body.

 Is there any log parsers( log4j/logback/Apache access log format)
 available in apache nifi?


 Any help on this much appreciated.

 Thanks in Advance.


>>>
>>
>

Re: Log4j/logback parser via syslog

[Joe Percivall]

Hello Madhu,


If you're looking for a template to show how to create a dynamic property for 
RouteOnAttribute to use, I'd suggest checking out this template[1]. It is a 
simple template that checks to see if the an attribute matches 'NiFi'.

Also provenance can be a very powerful debugging tool. If a flowfile gets 
routed to a relationship you don't expect, simply check the provenance for the 
destination of the relationship. You'll be able to see the exact attributes for 
any recent flowfile that was routed there.
[1] 
https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/simple-httpget-route-flow.xml

 
Hope that helps,
Joe

- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joeperciv...@yahoo.com



On Friday, February 12, 2016 2:28 PM, Madhukar Thota  
wrote:



I am getting my log4j logs on facility value 23 ( LOCAL7) how can route only 
facility 23 logs for further extraction.

I added RouteonAttribute  processor and defined this property 
:${facility:contains(23)}  but none of them messages getting matched. I am not 
sure my defined property is correct. How can i route messages based on the 
field value to different processors?

-Madhu


On Fri, Feb 12, 2016 at 11:33 AM, Madhukar Thota  
wrote:

Thanks Bryan. Looking forward for the release.
>
>
> 
>
>
>On Fri, Feb 12, 2016 at 10:55 AM, Bryan Bende  wrote:
>
>I believe groovy, python, jython, jruby, ruby, javascript, and lua.
>>
>>
>>The associated JIRA is here:
>>https://issues.apache.org/jira/browse/NIFI-210
>>
>>
>>
>>There are some cool blogs about them here:
>>http://funnifi.blogspot.com/2016/02/executescript-processor-hello-world.html
>>
>>
>>
>>-Bryan
>>
>>
>>On Fri, Feb 12, 2016 at 10:48 AM, Madhukar Thota  
>>wrote:
>>
>>Thanks Bryan. I will look into ExtractText processor.
>>>
>>>
>>>Do you know what scripting languages are supported with new processors?
>>>
>>>
>>>-Madhu
>>>
>>>
>>>On Fri, Feb 12, 2016 at 9:27 AM, Bryan Bende  wrote:
>>>
>>>Hello,


Currently there are no built in processors to parse log formats, but have 
you taken a look at the ExtractText processor [1]? 


If you can come up with a regular expression for whatever you are trying to 
extract, then you should be able to use ExtractText.


Other options... 


You could write a custom processor, but this sounds like it might be 
overkill for your scenario.
In the next release (hopefully out in a few days) there will be two new 
processors that support scripting languages. It may be easier to use a 
scripting language to manipulate/parse the text. 


Thanks,


Bryan


[1] 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ExtractText/index.html




On Fri, Feb 12, 2016 at 12:16 AM, Madhukar Thota  
wrote:

Hi 
>
>
>I am very new to Apache Nifi and just started learning about how to use it.
>
>
>We have a requirement where we need to parse log4j/logback pattern 
>messages coming from SyslogAppenders via Syslog udp. I can read the 
>standard syslog messages, but how can i further extract log4j/logback 
>messages  from syslog body.
>
>
>Is there any log parsers( log4j/logback/Apache access log format) 
>available in apache nifi?
>
>
>
>
>Any help on this much appreciated. 
>
>
>Thanks in Advance.
>
>

>>>
>>
>