Re: Drop events from Metron parser
Nifi’s Syslog 5424 support is based on the same library as Metron uses. On May 5, 2020 at 22:02:11, Dima Kovalyov (dimdr...@gmail.com) wrote: Hello Tom, Exactly, NiFi has range of ingest capable processors including Syslog server. - Dima On Tue, May 5, 2020, 20:00 Yerex, Tom wrote: > Hi Dima, > > Thanks for this. I have some knowledge of Nifi, but I'm still early on the > learning curve. > > Our current implementation plan is to use a collection of pre-existing log > servers and feed that into a Kafka cluster. In the model you describe would > that mean inserting NIfi between the log servers and Kafka? > > Cheers, > > Tom. > > > On 2020-05-05 17:25:01-07:00 Dima Kovalyov wrote: > > I would drop them on ingestion using NiFi's RouteOnContent. > > On Tue, May 5, 2020, 17:53 Yerex, Tom wrote: > >> Good afternoon, >> >> Our incoming data is not always perfect, in some cases events are simply >> missing fields. We would like a way to drop events when particular fields >> are empty (or have values we don't care about). >> >> One way we thought to do this might be to write a custom Stellar >> function. Does anyone know of another solution? >> >> Thank you, >> >> Tom. >> > - Dima > >
Re: Drop events from Metron parser
Hello Tom, Exactly, NiFi has range of ingest capable processors including Syslog server. - Dima On Tue, May 5, 2020, 20:00 Yerex, Tom wrote: > Hi Dima, > > Thanks for this. I have some knowledge of Nifi, but I'm still early on the > learning curve. > > Our current implementation plan is to use a collection of pre-existing log > servers and feed that into a Kafka cluster. In the model you describe would > that mean inserting NIfi between the log servers and Kafka? > > Cheers, > > Tom. > > > On 2020-05-05 17:25:01-07:00 Dima Kovalyov wrote: > > I would drop them on ingestion using NiFi's RouteOnContent. > > On Tue, May 5, 2020, 17:53 Yerex, Tom wrote: > >> Good afternoon, >> >> Our incoming data is not always perfect, in some cases events are simply >> missing fields. We would like a way to drop events when particular fields >> are empty (or have values we don't care about). >> >> One way we thought to do this might be to write a custom Stellar >> function. Does anyone know of another solution? >> >> Thank you, >> >> Tom. >> > - Dima > >
RE: Drop events from Metron parser
Hi Dima, Thanks for this. I have some knowledge of Nifi, but I'm still early on the learning curve. Our current implementation plan is to use a collection of pre-existing log servers and feed that into a Kafka cluster. In the model you describe would that mean inserting NIfi between the log servers and Kafka? Cheers, Tom. On 2020-05-05 17:25:01-07:00 Dima Kovalyov wrote: I would drop them on ingestion using NiFi's RouteOnContent. On Tue, May 5, 2020, 17:53 Yerex, Tom mailto:tom.ye...@ubc.ca>> wrote: Good afternoon, Our incoming data is not always perfect, in some cases events are simply missing fields. We would like a way to drop events when particular fields are empty (or have values we don't care about). One way we thought to do this might be to write a custom Stellar function. Does anyone know of another solution? Thank you, Tom. - Dima
Re: Drop events from Metron parser
I would drop them on ingestion using NiFi's RouteOnContent. On Tue, May 5, 2020, 17:53 Yerex, Tom wrote: > Good afternoon, > > Our incoming data is not always perfect, in some cases events are simply > missing fields. We would like a way to drop events when particular fields > are empty (or have values we don't care about). > > One way we thought to do this might be to write a custom Stellar function. > Does anyone know of another solution? > > Thank you, > > Tom. > - Dima
Re: Drop events from Metron parser
At the parser level, there's some configuration you can use for filtering events. Specifically "filterClassName". Take a look at the documentation, you can either use a custom class, or use Stellar. The example is even for "exists(field)", which you could modify to fail for missing fields. https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html On Tue, May 5, 2020 at 7:53 PM Yerex, Tom wrote: > Good afternoon, > > Our incoming data is not always perfect, in some cases events are simply > missing fields. We would like a way to drop events when particular fields > are empty (or have values we don't care about). > > One way we thought to do this might be to write a custom Stellar function. > Does anyone know of another solution? > > Thank you, > > Tom. >