Re: How can I ExtractGrok from end-of-file?
Hi Koji, thanks for the tip. I've ended writing a processor script for this task. Since I need the footer to validate the entire file, It was easier than splitting the files for the sole purpose of finding out the last line. Regards, Em qua, 20 de mar de 2019 às 00:59, Koji Kawamura escreveu: > Hello Eric, > > Have you found any solution for this? > If your trailers (footer?) starts with certain byte sequence, then > SplitContent may be helpful to split the content into Header+Payload, > and the Trailers. > If that works, then the subsequent flow can do something creative > probably using RouteOnAttribute, GrokExtract, MergeContent (with > defragment merge strategy) ... etc. > > Thanks, > Koji > > On Fri, Mar 15, 2019 at 11:34 PM Eric Chaves wrote: > > > > Hi folks, > > > > I'm learning how to use grok with nifi starting with the ExtractGrok > processor and I'd like to use it to extract data from file headers and > trailers however since the GrokExtract processor only apply the grok > expression on the defined buffer size (and each of my file differs on size) > I can't evaluate trailers on every file. > > > > Any ideas on how could I apply the grok expression from the end of file > instead of from the beginning, or any alternative processor? > > > > Cheers, > > >
Re: How can I ExtractGrok from end-of-file?
Hello Eric, Have you found any solution for this? If your trailers (footer?) starts with certain byte sequence, then SplitContent may be helpful to split the content into Header+Payload, and the Trailers. If that works, then the subsequent flow can do something creative probably using RouteOnAttribute, GrokExtract, MergeContent (with defragment merge strategy) ... etc. Thanks, Koji On Fri, Mar 15, 2019 at 11:34 PM Eric Chaves wrote: > > Hi folks, > > I'm learning how to use grok with nifi starting with the ExtractGrok > processor and I'd like to use it to extract data from file headers and > trailers however since the GrokExtract processor only apply the grok > expression on the defined buffer size (and each of my file differs on size) I > can't evaluate trailers on every file. > > Any ideas on how could I apply the grok expression from the end of file > instead of from the beginning, or any alternative processor? > > Cheers, >
How can I ExtractGrok from end-of-file?
Hi folks, I'm learning how to use grok with nifi starting with the ExtractGrok processor and I'd like to use it to extract data from file headers and trailers however since the GrokExtract processor only apply the grok expression on the defined buffer size (and each of my file differs on size) I can't evaluate trailers on every file. Any ideas on how could I apply the grok expression from the end of file instead of from the beginning, or any alternative processor? Cheers,