Andre, See here https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
I just added an example template and some instructions to get you started. This should really help you nail down the concept and think of ways to leverage it in your flows and what things you'd want to offer out of the box as far as readers/writers/etc. Thanks Joe On Sat, May 6, 2017 at 4:03 AM, Andre <[email protected]> wrote: > Joe, > > My question wasn't very clear but I think you still managed to provide the > insight I needed! > > I will give it a go creating a reader to see if I get a better view on how > those work. > > Cheers > > On Sat, May 6, 2017 at 1:22 PM, Joe Witt <[email protected]> wrote: > >> Andre >> >> If I understand the question correctly the ask is how can you extend >> the existing set of readers and writers. This is accomplished by >> implementing controller services. You can build them in Java or you >> can build them using the scripting language support for record >> readers/writers. >> >> You'd want a reader understands how to deserialize the bytes of a CEF >> formatted file into record objects and a writer that understands how >> to take record objects and serialize them into CEF. Same for evtx. >> As you deserialize and parse the events you identify fields and values >> and create the record structure. >> >> Any of the processors that support the record concept have a reader >> and writer configured on them. If you wanted to end up with JSON >> records then you'd plug-in a json writer. You can get a sense of much >> of this from unit tests and I'd also be happy to put up a template >> that shows off how to configure a flow that reads from NiFi's own >> provenance stream, turns it into JSON, Avro, plaintext, and xml all at >> once. I'll try to do that tomorrow. That should make it a lot more >> clear. >> >> Thanks >> >> On Fri, May 5, 2017 at 10:51 PM, Andre <[email protected]> wrote: >> > All, >> > >> > I was doing some reading during my spare time and the record related >> > feature set is truly exciting. >> > >> > However I was wondering, how should one extend the existing range of >> > readers? >> > >> > I use for example ParseCEF and ParseEvtx. CEF while not particularly >> simple >> > to parse and validate using RegEx'es (and Grok) is a reasonably >> structured >> > format (with the same field supporting 9 different date formats...) >> > >> > ParseCEF currently exposes the parsed and validade CEF payload either to >> > attributes or JSON to content file. And hence my question: >> > >> > Is this the desired path of action: >> > >> > ParseCEF -> JSON content -> ConvertRecord (using JsonPathReader) -> >> whatever >> > >> > ? >> > >> > Or should extension make more sense? >> > >> > >> > Cheers >>
