Re: Drop events from Metron parser

2020-05-11 Thread Otto Fowler
Nifi’s Syslog 5424 support is based on the same library as Metron uses. On May 5, 2020 at 22:02:11, Dima Kovalyov (dimdr...@gmail.com) wrote: Hello Tom, Exactly, NiFi has range of ingest capable processors including Syslog server. - Dima On Tue, May 5, 2020, 20:00 Yerex, Tom wrote: > Hi

Re: Drop events from Metron parser

2020-05-05 Thread Dima Kovalyov
Hello Tom, Exactly, NiFi has range of ingest capable processors including Syslog server. - Dima On Tue, May 5, 2020, 20:00 Yerex, Tom wrote: > Hi Dima, > > Thanks for this. I have some knowledge of Nifi, but I'm still early on the > learning curve. > > Our current implementation plan is to

RE: Drop events from Metron parser

2020-05-05 Thread Yerex, Tom
Hi Dima, Thanks for this. I have some knowledge of Nifi, but I'm still early on the learning curve. Our current implementation plan is to use a collection of pre-existing log servers and feed that into a Kafka cluster. In the model you describe would that mean inserting NIfi between the log

Re: Drop events from Metron parser

2020-05-05 Thread Dima Kovalyov
I would drop them on ingestion using NiFi's RouteOnContent. On Tue, May 5, 2020, 17:53 Yerex, Tom wrote: > Good afternoon, > > Our incoming data is not always perfect, in some cases events are simply > missing fields. We would like a way to drop events when particular fields > are empty (or

Re: Drop events from Metron parser

2020-05-05 Thread Justin Leet
At the parser level, there's some configuration you can use for filtering events. Specifically "filterClassName". Take a look at the documentation, you can either use a custom class, or use Stellar. The example is even for "exists(field)", which you could modify to fail for missing fields.