Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Otto Fowler
I’ll put some thoughts in METRON-1453, unless we want a discuss thread On July 20, 2018 at 10:32:48, Casey Stella (ceste...@gmail.com) wrote: So, I would really love to see METRON-1453 go in, because I'd love to decouple syslog parsing (very common) from generic grok. On Fri, Jul 20, 2018 at

Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Casey Stella
So, I would really love to see METRON-1453 go in, because I'd love to decouple syslog parsing (very common) from generic grok. On Fri, Jul 20, 2018 at 10:26 AM Otto Fowler wrote: > Metron does not have a generic Syslog Parser. > > Nifi has Syslog parsing ( either Records or standard Processor

Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Otto Fowler
Metron does not have a generic Syslog Parser. Nifi has Syslog parsing ( either Records or standard Processor ), in two modes. ParseSyslog is the original, where regex’s are used to parse the syslog RFC3164 and RFC5424, but only extracts the common fields ( so the ‘additional info’ like program

Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Casey Stella
I just want to pile in here and recommend taking a look at the parser chaining use-case, which is a walk-through of pulling in firewall logs over syslog using grok ( https://github.com/apache/metron/tree/master/use-cases/parser_chaining). Unfortunately this is in master and yet in a release, but

Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Simon Elliston Ball
What you need to do is NOT ParseCEF in NiFi. Metron should handle be CEF parsing. Just use NiFi to do the listen syslog (no need to parse in NiFi) then SplitText to get one line of CEF per kafka message (if your syslog is batching, this may not be necessary. Set up a sensor in Metron using

Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Srikanth Nagarajan
Hi Farrukh,You can try using the Grok Parser and search for regular _expression_ pattern for your log.  You can customize the regex to meet your needs.   https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+TelemetryLook at Step-5

CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Farrukh Naveed Anjum
Hi, I am trying to index the Syslog using CEF Parser with Nifi. It does not give any error though, transport data to kafa without indexing it. It keepg giving FAILED in Spout. I believe indexing Syslog are most basic usecase for all. But metron fails to do it with each in standard format. I